Jul 29 01:08:07.501243 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 01:08:07.501256 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 01:08:07.501266 ip-10-0-136-251 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 01:08:07.501603 ip-10-0-136-251 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 01:08:17.586637 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 01:08:17.586652 ip-10-0-136-251 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6b9e8e88fa9441f9a54d85b723b51e5a --
Jul 29 01:10:12.708306 ip-10-0-136-251 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 01:10:13.151372 ip-10-0-136-251 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 01:10:13.151372 ip-10-0-136-251 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 01:10:13.151372 ip-10-0-136-251 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 01:10:13.151372 ip-10-0-136-251 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 01:10:13.151372 ip-10-0-136-251 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 01:10:13.152053 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.151449    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 01:10:13.155077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155062    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.155077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155076    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155081    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155086    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155089    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155092    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155095    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155098    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155101    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155104    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155107    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155109    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155112    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155114    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155117    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155120    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155123    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155125    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155128    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155131    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.155139 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155133    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155136    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155138    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155144    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155147    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155150    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155153    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155155    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155158    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155160    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155163    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155165    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155168    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155170    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155173    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155176    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155180    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155183    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155186    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.155588 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155188    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155191    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155194    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155197    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155200    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155202    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155205    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155208    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155211    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155214    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155216    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155219    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155221    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155224    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155226    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155230    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155232    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155235    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155237    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155240    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.156103 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155242    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155245    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155247    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155250    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155252    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155255    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155257    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155260    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155262    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155265    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155268    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155271    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155274    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155276    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155279    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155281    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155284    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155286    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155290    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155294    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.156582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155296    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155300    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155302    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155305    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155308    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155310    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155314    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155317    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155319    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155322    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155324    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155327    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155329    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155332    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155335    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155338    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155340    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155343    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155345    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.157070 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.155348    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156059    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156066    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156069    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156072    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156075    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156078    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156081    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156084    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156087    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156089    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156092    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156094    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156097    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156100    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156103    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156108    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156112    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156115    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156118    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.157517 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156122    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156126    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156128    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156131    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156134    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156136    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156139    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156141    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156144    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156147    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156149    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156153    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156156    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156158    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156161    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156164    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156166    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156170    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156172    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156175    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.158117 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156177    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156180    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156182    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156185    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156187    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156190    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156193    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156195    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156200    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156202    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156205    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156208    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156211    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156214    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156216    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156219    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156221    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156225    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156227    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.158657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156230    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156233    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156235    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156239    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156243    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156246    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156248    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156251    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156253    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156256    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156259    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156261    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156264    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156266    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156269    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156271    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156273    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156276    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156278    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.159136 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156281    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156283    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156286    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156289    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156291    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156294    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156297    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156300    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156302    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156305    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156307    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156310    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156312    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156314    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156317    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156319    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156322    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156324    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156327    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156329    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.159582 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156332    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156334    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156727    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156732    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156735    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156738    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156741    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156743    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156746    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156748    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156751    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156754    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156756    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156759    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156762    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156767    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156770    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156774    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156777    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.160085 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156780    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156784    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156786    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156790    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156793    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156796    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156798    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156801    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156803    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156806    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156808    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156811    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156814    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156816    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156819    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156821    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156824    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156827    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156829    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156831    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.160542 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156834    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156836    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156838    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156841    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156844    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156846    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156848    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156851    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156854    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156857    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156860    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156863    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156865    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156868    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156870    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156873    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156875    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156877    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156880    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156882    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.161017 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156885    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156887    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156890    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156892    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156894    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156897    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156899    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156902    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156904    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156907    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156909    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156911    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156914    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156916    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156920    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156923    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156926    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156929    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156931    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.161537 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156934    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156936    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156939    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156942    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156944    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156947    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156950    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156952    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156954    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156957    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156959    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156961    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156964    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156966    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156969    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156971    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156974    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156977    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156980    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156983    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.161998 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156985    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156988    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.156990    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157069    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157075    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157092    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157097    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157101    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157104    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157108    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157112    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157115    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157118    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157121    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157125    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157128    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157134    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157137    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157140    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157143    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157146    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157150    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157153    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 01:10:13.162483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157157    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157160    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157163    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157166    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157169    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157172    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157175    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157178    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157181    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157184    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157187    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157190    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157194    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157197    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157200    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157203    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157206    2577 flags.go:64] FLAG: --enable-server="true"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157208    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157212    2577 flags.go:64] FLAG: --event-burst="100"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157215    2577 flags.go:64] FLAG: --event-qps="50"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157219    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157222    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157225    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157228    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 01:10:13.163056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157231    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157236    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157239    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157242    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157245    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157248    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157251    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157254    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157257    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157260    2577 flags.go:64] FLAG: --feature-gates=""
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157263    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157266    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157269    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157272    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157275    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157278    2577 flags.go:64] FLAG: --help="false"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157281    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157284    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157287    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157290    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157293    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157296    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157299    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 01:10:13.163623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157302    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157304    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157307    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157311    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157314    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157318    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157321    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157324    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157327    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157330    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157334    2577 flags.go:64] FLAG: --lock-file=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157337    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157340    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157343    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157348    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157351    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157354    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157357    2577 flags.go:64] FLAG: --logging-format="text"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157360    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157363    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157366    2577 flags.go:64] FLAG: --manifest-url=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157369    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157373    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157376    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157381    2577 flags.go:64] FLAG: --max-pods="110"
Jul 29 01:10:13.164252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157384    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157387    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157390    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157392    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157395    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157398    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157401    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157407    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157410    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157413    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157415    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157418    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157424    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157427    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157430    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157432    2577 flags.go:64] FLAG: --port="10250"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157435    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157438    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-030d14721d2e34687"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157443    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157446    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157449    2577 flags.go:64] FLAG: --register-node="true"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157452    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157456    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157459    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 29 01:10:13.164844 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157461    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157464    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157468    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157471    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157474    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157477    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157480    2577 flags.go:64] FLAG: --runonce="false"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157482    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157485    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157488    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157491    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157494    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157497    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157500    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157503    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157505    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157508    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157511    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157514    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157517    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157524    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157527    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157532    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157535    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157538    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 01:10:13.165466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157580    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157585    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157588    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157592    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157595    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157598    2577 flags.go:64] FLAG: --v="2"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157602    2577 flags.go:64] FLAG: --version="false"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157606    2577 flags.go:64] FLAG: --vmodule=""
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157610    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.157613    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157733    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157737    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157740    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157743    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157746    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157749    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157751    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157754    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157757    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157760    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157763    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157766    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.166077 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157769    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157772    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157775    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157778    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157782    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157787    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157790    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157793    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157796    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157798    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157801    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157804    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157807    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157810    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157812    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157815    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157818    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157821    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157823    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.166599 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157827    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157830    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157833    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157836    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157838    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157841    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157843    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157846    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157848    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157851    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157854    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157856    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157859    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157861    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157864    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157866    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157869    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157871    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157875    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157877    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.167079 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157880    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157883    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157885    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157888    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157893    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157895    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157898    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157900    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157903    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157906    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157909    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157911    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157914    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157917    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157919    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157922    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157924    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157927    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157929    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157932    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.167604 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157934    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157937    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157939    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157942    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157944    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157947    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157950    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157954    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157957    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157960    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157964    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157967    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157969    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157972    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157974    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157977    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157981    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157983    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157986    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.168109 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157988    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157991    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157994    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157997    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.157999    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.158002    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.158004    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.158007    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.158009    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.158095    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.165555    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.165572    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.165586    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.165596    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 01:10:13.168572 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165651    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165656    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165659    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165662    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165665    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165668    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165671    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165673    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165676    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165679    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165681    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165683    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165686    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165689    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165691    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165694    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165696    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165699    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165702    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.168930 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165706    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165709    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165712    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165715    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165718    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165720    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165723    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165726    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165729    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165731    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165735    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165737    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165740    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165743    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165746    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165749    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165751    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165754    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165756    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165759    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.169460 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165761    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165764    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165766    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165770    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165774    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165777    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165779    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165782    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165784    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165787    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165790    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165792    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165795    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165797    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165800    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165803    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165805    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165808    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165811    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.169940 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165813    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165816    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165819    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165822    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165825    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165827    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165830    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165833    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165835    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165837    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165840    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165842    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165845    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165847    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165850    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165853    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165855    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165857    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165860    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165862    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.170407 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165866    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165868    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165871    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165873    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165876    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165878    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165881    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165883    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165886    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165888    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165890    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165895    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165898    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165901    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165903    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165906    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165909    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165913    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165916    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.170919 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165918    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.165921    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.165926    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166055    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166062    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166064    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166067    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166070    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166073    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166076    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166078    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166081    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166084    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166086    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166089    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 01:10:13.171388 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166092    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166096    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166099    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166101    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166104    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166106    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166109    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166111    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166114    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166116    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166119    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166121    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166124    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166126    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166129    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166131    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166135    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166137    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166140    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166142    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 01:10:13.171755 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166145    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166147    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166150    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166152    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166155    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166158    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166160    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166163    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166166    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166168    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166171    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166173    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166176    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166178    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166181    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166183    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166186    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166188    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166190    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 01:10:13.172251 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166193    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166195    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166198    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166200    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166203    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166205    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166208    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166210    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166213    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166215    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166218    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166221    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166223    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166226    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166229    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166233    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166237    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166240    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166243    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 01:10:13.172712 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166246    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166248    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166251    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166255    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166258    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166261    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166263    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166266    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166268    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166271    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166273    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166276    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166278    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166280    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166283    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166285    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166288    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166290    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166293    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166295    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 01:10:13.173228 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166299    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166301    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166303    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166306    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166309    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166311    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166314    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166316    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:13.166318    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.166323    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.166424    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.168851    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.170208    2577 server.go:1019] "Starting client certificate rotation"
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.170310    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 01:10:13.173706 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.171418    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 01:10:13.174130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.172523    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 01:10:13.174130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.174057    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 01:10:13.186797 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.186776    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 29 01:10:13.188726 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.188711    2577 log.go:25] "Validated CRI v1 image API"
Jul 29 01:10:13.189492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.189473    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 01:10:13.193221 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.193203    2577 fs.go:135] Filesystem UUIDs: map[2b0bce3f-77c5-416c-99f6-da947a3e5be0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c3154879-9e54-462e-87ae-0bdf62c673b5:/dev/nvme0n1p4]
Jul 29 01:10:13.193302 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.193220    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 01:10:13.199763 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.199648    2577 manager.go:216] Machine: {Timestamp:2026-07-29 01:10:13.1983144 +0000 UTC m=+0.351759767 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098791 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2260f7b9885aec0aadca12f45c4a6c SystemUUID:ec2260f7-b988-5aec-0aad-ca12f45c4a6c BootID:6b9e8e88-fa94-41f9-a54d-85b723b51e5a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d6:6a:a0:7f:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d6:6a:a0:7f:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:3d:ad:d3:d1:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 01:10:13.199763 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.199749    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 01:10:13.199901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.199828    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 01:10:13.200243 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.200226    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 01:10:13.202880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.202853    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 01:10:13.203005 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.202882    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 01:10:13.204110 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.204099    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 01:10:13.204144 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.204113    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 01:10:13.204144 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.204120    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 01:10:13.204856 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.204846    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 01:10:13.205536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.205527    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 01:10:13.207156 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207145    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 01:10:13.207269 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207259    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 01:10:13.207306 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207292    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 01:10:13.207306 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207306    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 01:10:13.207363 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207319    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 01:10:13.207363 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207328    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 29 01:10:13.207363 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207337    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 01:10:13.207649 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207629    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 01:10:13.207734 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.207654    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 01:10:13.210510 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.210494    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 01:10:13.212443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.212431    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 01:10:13.212500 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.212460    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 01:10:13.213982 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.213970    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 01:10:13.214032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.213990    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 01:10:13.214032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.213999    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 01:10:13.214032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214007    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 01:10:13.214032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214019    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 01:10:13.214032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214028    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214053    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214062    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214071    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214079    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214091    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 01:10:13.214184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.214099    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 01:10:13.215938 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.215926    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 01:10:13.215938 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.215939    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 01:10:13.216156 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.216135    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 01:10:13.216230 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.216199    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 01:10:13.218873 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.218861    2577 server.go:1268] "Started kubelet"
Jul 29 01:10:13.218991 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.218961    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 01:10:13.219058 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.218966    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 01:10:13.219058 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.219034    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 01:10:13.219770 ip-10-0-136-251 systemd[1]: Started Kubernetes Kubelet.
Jul 29 01:10:13.220488 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.220475    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 01:10:13.220887 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.220871    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vc42p"
Jul 29 01:10:13.224516 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.224491    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 01:10:13.224516 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.224499    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 01:10:13.228440 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.228422    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 01:10:13.229099 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229080    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 01:10:13.229185 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229125    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 01:10:13.229185 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.229179    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.229335 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.228352    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-251.ec2.internal.18c69d7186e26e32  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-251.ec2.internal,UID:ip-10-0-136-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-251.ec2.internal,},FirstTimestamp:2026-07-29 01:10:13.218831922 +0000 UTC m=+0.372277288,LastTimestamp:2026-07-29 01:10:13.218831922 +0000 UTC m=+0.372277288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-251.ec2.internal,}"
Jul 29 01:10:13.229629 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229612    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 01:10:13.229629 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229614    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 01:10:13.229777 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229639    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 01:10:13.229777 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229754    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 01:10:13.229777 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.229763    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 01:10:13.230234 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230185    2577 factory.go:153] Registering CRI-O factory
Jul 29 01:10:13.230234 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230204    2577 factory.go:223] Registration of the crio container factory successfully
Jul 29 01:10:13.230366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230252    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 01:10:13.230366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230262    2577 factory.go:55] Registering systemd factory
Jul 29 01:10:13.230366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230268    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 29 01:10:13.230366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230295    2577 factory.go:103] Registering Raw factory
Jul 29 01:10:13.230366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230309    2577 manager.go:1196] Started watching for new ooms in manager
Jul 29 01:10:13.230873 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.230705    2577 manager.go:349] Starting recovery of all containers
Jul 29 01:10:13.232971 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.232948    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vc42p"
Jul 29 01:10:13.233317 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.233268    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 01:10:13.234157 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.234111    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 01:10:13.241771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.241756    2577 manager.go:354] Recovery completed
Jul 29 01:10:13.246001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.245988    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.250546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.250531    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.250622 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.250564    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.250622 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.250578    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.251093 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251075    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.251159 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251074    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 01:10:13.251159 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251131    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 01:10:13.251159 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251146    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 01:10:13.251817 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251795    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.251863 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251833    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.251863 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.251847    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.252573 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.252562    2577 policy_none.go:49] "None policy: Start"
Jul 29 01:10:13.252604 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.252577    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 01:10:13.252604 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.252587    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 01:10:13.253527 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.253518    2577 policy_none.go:47] "Start"
Jul 29 01:10:13.287845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.287296    2577 manager.go:337] "Starting Device Plugin manager"
Jul 29 01:10:13.287845 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.287529    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 01:10:13.287845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.287547    2577 server.go:87] "Starting device plugin registration server"
Jul 29 01:10:13.287845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.287791    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 01:10:13.288104 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.287804    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 01:10:13.288104 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.287929    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 01:10:13.288104 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.288009    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 01:10:13.288104 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.288020    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 01:10:13.289211 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.289189    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 01:10:13.289289 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.289241    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.390676 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.390654    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.391731 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.391693    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.391731 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.391720    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.391731 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.391732    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.391859 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.391756    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.400286 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.400269    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.400340 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.400289    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-251.ec2.internal\": node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.416465 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.416408    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.458743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.458716    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 01:10:13.460027 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.460008    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 01:10:13.460134 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.460111    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 01:10:13.460196 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.460151    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 01:10:13.460254 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.460196    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 01:10:13.462332 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.462314    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:13.518739 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.518710    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.562092 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.562037    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"]
Jul 29 01:10:13.562186 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.562175    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.563076 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.563060    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.563150 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.563089    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.563150 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.563100    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.564310 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.564298    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.564495 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.564480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.564546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.564509    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.565044 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565026    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.565148 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565034    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.565148 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565070    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.565148 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565081    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.565148 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565071    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.565280 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.565154    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.566283 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.566265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.566383 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.566290    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 01:10:13.566978 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.566954    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 01:10:13.567083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.566985    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 01:10:13.567083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.566997    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID"
Jul 29 01:10:13.585684 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.585661    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-251.ec2.internal\" not found" node="ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.590236 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.590221    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-251.ec2.internal\" not found" node="ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.618963 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.618928    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.633783 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.633760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.633838 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.633789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.633838 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.633807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee83001d783d1446eea0c77a1a727ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"1ee83001d783d1446eea0c77a1a727ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.719830 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.719767    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.734225 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.734285 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.734285 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.734351 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee83001d783d1446eea0c77a1a727ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"1ee83001d783d1446eea0c77a1a727ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.734351 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee83001d783d1446eea0c77a1a727ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"1ee83001d783d1446eea0c77a1a727ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.734417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.734349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1421b2f48a6ec546abab4c3baa57eb62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"1421b2f48a6ec546abab4c3baa57eb62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.826354 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.826324    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:13.888544 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.888513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.894382 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:13.894361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:13.926735 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:13.926701    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:14.030760 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.030650    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found"
Jul 29 01:10:14.037520 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.037499    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:14.129916 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.129872    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:14.139808 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.139781    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:14.142731 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.142717    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 01:10:14.144165 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.144151    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"
Jul 29 01:10:14.153113 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.153099    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 01:10:14.170521 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.170501    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 01:10:14.170653 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.170633    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 01:10:14.170701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.170664    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 01:10:14.170701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.170664    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 01:10:14.208019 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.207991    2577 apiserver.go:52] "Watching apiserver"
Jul 29 01:10:14.227271 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.227243    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 01:10:14.228253 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.228228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-hlw75","openshift-network-diagnostics/network-check-target-hsrgh","openshift-network-operator/iptables-alerter-2bxkz","openshift-ovn-kubernetes/ovnkube-node-dfrwb","kube-system/global-pull-secret-syncer-qdhzb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc","openshift-cluster-node-tuning-operator/tuned-hfxjm","openshift-dns/node-resolver-xsfjt","openshift-image-registry/node-ca-mfgmj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal","openshift-multus/multus-9nz88","openshift-multus/network-metrics-daemon-ctvcr","kube-system/konnectivity-agent-bj86g","kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"]
Jul 29 01:10:14.229483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.229464    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 01:10:14.229714 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.229697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.230984 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.230957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.232043 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.232027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.232436 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.232417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.232522 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.232444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kq67x\""
Jul 29 01:10:14.232647 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.232417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.233273 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.233256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.234238 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.234218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 01:10:14.234924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.234494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.234924 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.234584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:14.236188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236165    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 01:10:14.236291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 01:10:14.236291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.236473 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l5f5k\""
Jul 29 01:10:14.236580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236556    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 01:10:14.236698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b688l\""
Jul 29 01:10:14.236875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.236964 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysconfig\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237038 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.236988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-tmp\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237131 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237052    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 01:05:13 +0000 UTC" deadline="2028-02-25 02:50:48.015197827 +0000 UTC"
Jul 29 01:10:14.237131 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa339f5a-ecb3-4616-8e1e-02dba66601d8-agent-certs\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.237215 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237132    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13825h40m33.778071432s"
Jul 29 01:10:14.237464 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa339f5a-ecb3-4616-8e1e-02dba66601d8-konnectivity-ca\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-conf\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-systemd\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-host\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q27s\" (UniqueName: \"kubernetes.io/projected/47e9a927-2417-42cb-9e94-f42270a9d4e1-kube-api-access-5q27s\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-run\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-lib-modules\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-modprobe-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-kubernetes\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-sys\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-var-lib-kubelet\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-tuned\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m72nq\""
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 01:10:14.237965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.237959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 01:10:14.238383 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.238156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.238971 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.238949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.240127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.240100    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.240217 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.240196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.240217 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.240203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 01:10:14.240905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.240890    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhnpr\""
Jul 29 01:10:14.240905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.240899    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.241379 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.241362    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.241379 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.241373    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.241608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.241399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 01:10:14.241608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.241412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vhg2c\""
Jul 29 01:10:14.241726 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.241712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:14.241789 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.241773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:14.242023 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.242003    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 01:10:14.242804 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.242784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.242804 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.242797    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2bckl\""
Jul 29 01:10:14.242921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.242830    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 01:10:14.243122 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.243078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.243553 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.243532    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 01:10:14.243647 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.243543    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.243829 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.243810    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 01:10:14.244366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.244348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.245452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.245424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 01:10:14.245598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.245580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 01:10:14.245694 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.245680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.245749 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.245732    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:14.245870 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.245855    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mnnn\""
Jul 29 01:10:14.246663 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.246647    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 01:10:14.246765 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.246747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5vqsc\""
Jul 29 01:10:14.266165 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.266149    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffwpc"
Jul 29 01:10:14.274843 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.274826    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffwpc"
Jul 29 01:10:14.331030 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.331006    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 01:10:14.338534 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-sys\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-netd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwc4\" (UniqueName: \"kubernetes.io/projected/e84ac7e9-e8a9-4357-aefc-48f48a65d883-kube-api-access-4zwc4\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-log-socket\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovn-node-metrics-cert\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-kubelet-config\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-sys\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-sys-fs\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48kc\" (UniqueName: \"kubernetes.io/projected/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-kube-api-access-c48kc\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.338744 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-conf\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-host\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q27s\" (UniqueName: \"kubernetes.io/projected/47e9a927-2417-42cb-9e94-f42270a9d4e1-kube-api-access-5q27s\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-conf\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-host\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-netns\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-env-overrides\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.338990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg96j\" (UniqueName: \"kubernetes.io/projected/bba7d09a-2198-469c-8971-19c969703358-kube-api-access-wg96j\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-run\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-os-release\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-run\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-cni-binary-copy\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-systemd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5v4\" (UniqueName: \"kubernetes.io/projected/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-kube-api-access-hl5v4\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-dbus\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0da695-b68c-4390-abd9-9bd1383c5e87-iptables-alerter-script\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fwq\" (UniqueName: \"kubernetes.io/projected/0a0da695-b68c-4390-abd9-9bd1383c5e87-kube-api-access-24fwq\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-tuned\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84ac7e9-e8a9-4357-aefc-48f48a65d883-hosts-file\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-bin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-multus-certs\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-registration-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-tmp\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa339f5a-ecb3-4616-8e1e-02dba66601d8-konnectivity-ca\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-multus\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-conf-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-bin\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-config\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-system-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.339691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339482    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-slash\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-kubernetes\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-cnibin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-ovn\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zf4s\" (UniqueName: \"kubernetes.io/projected/58b72d20-daee-435f-a08b-856037109537-kube-api-access-9zf4s\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-kubernetes\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-kubelet\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa339f5a-ecb3-4616-8e1e-02dba66601d8-konnectivity-ca\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysctl-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/0c7f0007-c5de-4625-a970-c636ae3ab383-kube-api-access-lrfgf\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.340423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.339998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-device-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-systemd-units\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-etc-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-serviceca\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-k8s-cni-cncf-io\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-hostroot\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-daemon-config\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-var-lib-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-host\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-system-cni-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-modprobe-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-var-lib-kubelet\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-socket-dir-parent\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-modprobe-d\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-etc-kubernetes\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-var-lib-kubelet\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-socket-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0da695-b68c-4390-abd9-9bd1383c5e87-host-slash\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysconfig\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa339f5a-ecb3-4616-8e1e-02dba66601d8-agent-certs\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-netns\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-sysconfig\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-node-log\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-cnibin\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-os-release\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-systemd\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84ac7e9-e8a9-4357-aefc-48f48a65d883-tmp-dir\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-systemd\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-lib-modules\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.341661 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6hf\" (UniqueName: \"kubernetes.io/projected/379367f7-b885-4486-b103-e9d470325865-kube-api-access-cs6hf\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.342181 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.340975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-kubelet\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.342181 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.341005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47e9a927-2417-42cb-9e94-f42270a9d4e1-lib-modules\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.342181 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.341007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-script-lib\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.342181 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.341039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.342181 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.341089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.342660 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.342629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-tmp\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.342770 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.342715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47e9a927-2417-42cb-9e94-f42270a9d4e1-etc-tuned\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.343236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.343218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa339f5a-ecb3-4616-8e1e-02dba66601d8-agent-certs\") pod \"konnectivity-agent-bj86g\" (UID: \"fa339f5a-ecb3-4616-8e1e-02dba66601d8\") " pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.346675 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.346658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q27s\" (UniqueName: \"kubernetes.io/projected/47e9a927-2417-42cb-9e94-f42270a9d4e1-kube-api-access-5q27s\") pod \"tuned-hfxjm\" (UID: \"47e9a927-2417-42cb-9e94-f42270a9d4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.434920 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.434886    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1421b2f48a6ec546abab4c3baa57eb62.slice/crio-23a76b722a5b6b87d2d496f07a29700fe03070a5b126a61b33bce5018a7d299d WatchSource:0}: Error finding container 23a76b722a5b6b87d2d496f07a29700fe03070a5b126a61b33bce5018a7d299d: Status 404 returned error can't find the container with id 23a76b722a5b6b87d2d496f07a29700fe03070a5b126a61b33bce5018a7d299d
Jul 29 01:10:14.440500 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.440480    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:10:14.442236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwc4\" (UniqueName: \"kubernetes.io/projected/e84ac7e9-e8a9-4357-aefc-48f48a65d883-kube-api-access-4zwc4\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.442305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-log-socket\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.442305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovn-node-metrics-cert\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.442305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-kubelet-config\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.442443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-sys-fs\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.442443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48kc\" (UniqueName: \"kubernetes.io/projected/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-kube-api-access-c48kc\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.442443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-log-socket\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.442684 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-sys-fs\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.442684 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-kubelet-config\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.442810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:14.442810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-netns\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.442810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-env-overrides\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.442810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg96j\" (UniqueName: \"kubernetes.io/projected/bba7d09a-2198-469c-8971-19c969703358-kube-api-access-wg96j\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.443016 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-os-release\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.443016 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-cni-binary-copy\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.443016 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-systemd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.443016 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.442994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-os-release\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.443253 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-netns\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.443253 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5v4\" (UniqueName: \"kubernetes.io/projected/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-kube-api-access-hl5v4\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.443253 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-dbus\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.443404 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-systemd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.443404 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-env-overrides\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d5adf746-cbd1-47d6-a225-636fdbc225c4-dbus\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0da695-b68c-4390-abd9-9bd1383c5e87-iptables-alerter-script\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-cni-binary-copy\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fwq\" (UniqueName: \"kubernetes.io/projected/0a0da695-b68c-4390-abd9-9bd1383c5e87-kube-api-access-24fwq\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84ac7e9-e8a9-4357-aefc-48f48a65d883-hosts-file\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-bin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-multus-certs\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-registration-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-multus\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-conf-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-bin\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-config\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-system-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-slash\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.444794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.443986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-cnibin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-ovn\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zf4s\" (UniqueName: \"kubernetes.io/projected/58b72d20-daee-435f-a08b-856037109537-kube-api-access-9zf4s\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a0da695-b68c-4390-abd9-9bd1383c5e87-iptables-alerter-script\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-kubelet\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/0c7f0007-c5de-4625-a970-c636ae3ab383-kube-api-access-lrfgf\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-device-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84ac7e9-e8a9-4357-aefc-48f48a65d883-hosts-file\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-bin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-multus-certs\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-registration-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-cni-multus\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.445812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-systemd-units\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-conf-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-etc-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-serviceca\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-bin\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-k8s-cni-cncf-io\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-k8s-cni-cncf-io\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.444215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-var-lib-kubelet\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.445299    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.445364    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.445372    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:14.945349413 +0000 UTC m=+2.098794769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.445420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:14.945403291 +0000 UTC m=+2.098848663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-system-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-config\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-cni-dir\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-device-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.446739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-slash\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovn-node-metrics-cert\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-cnibin\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-systemd-units\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-hostroot\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-daemon-config\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-var-lib-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-host\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-system-cni-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-socket-dir-parent\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-etc-kubernetes\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.445992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-socket-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0da695-b68c-4390-abd9-9bd1383c5e87-host-slash\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-netns\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-serviceca\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-node-log\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.447565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-hostroot\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-etc-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-cnibin\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-os-release\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-host\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-var-lib-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84ac7e9-e8a9-4357-aefc-48f48a65d883-tmp-dir\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-host-run-netns\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6hf\" (UniqueName: \"kubernetes.io/projected/379367f7-b885-4486-b103-e9d470325865-kube-api-access-cs6hf\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-node-log\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-kubelet\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-script-lib\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-socket-dir-parent\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c7f0007-c5de-4625-a970-c636ae3ab383-etc-kubernetes\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-netd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bba7d09a-2198-469c-8971-19c969703358-socket-dir\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c7f0007-c5de-4625-a970-c636ae3ab383-multus-daemon-config\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84ac7e9-e8a9-4357-aefc-48f48a65d883-tmp-dir\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-cni-netd\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a0da695-b68c-4390-abd9-9bd1383c5e87-host-slash\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.446795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-openvswitch\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-kubelet\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-run-ovn\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-cnibin\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-os-release\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b72d20-daee-435f-a08b-856037109537-system-cni-dir\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.448888 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.449443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58b72d20-daee-435f-a08b-856037109537-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.449443 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.447807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-ovnkube-script-lib\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.450827 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.450789    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:14.450827 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.450818    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:14.450827 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.450834    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:14.451076 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.450906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:14.950888957 +0000 UTC m=+2.104334336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:14.453287 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.453217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5v4\" (UniqueName: \"kubernetes.io/projected/7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27-kube-api-access-hl5v4\") pod \"ovnkube-node-dfrwb\" (UID: \"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.453287 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.453234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48kc\" (UniqueName: \"kubernetes.io/projected/bae9cb32-dcf4-4464-b83f-77ad30c84c7a-kube-api-access-c48kc\") pod \"node-ca-mfgmj\" (UID: \"bae9cb32-dcf4-4464-b83f-77ad30c84c7a\") " pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.453643 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.453619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg96j\" (UniqueName: \"kubernetes.io/projected/bba7d09a-2198-469c-8971-19c969703358-kube-api-access-wg96j\") pod \"aws-ebs-csi-driver-node-qdtmc\" (UID: \"bba7d09a-2198-469c-8971-19c969703358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.453726 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.453672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/0c7f0007-c5de-4625-a970-c636ae3ab383-kube-api-access-lrfgf\") pod \"multus-9nz88\" (UID: \"0c7f0007-c5de-4625-a970-c636ae3ab383\") " pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.453778 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.453732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwc4\" (UniqueName: \"kubernetes.io/projected/e84ac7e9-e8a9-4357-aefc-48f48a65d883-kube-api-access-4zwc4\") pod \"node-resolver-xsfjt\" (UID: \"e84ac7e9-e8a9-4357-aefc-48f48a65d883\") " pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.454268 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.454247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fwq\" (UniqueName: \"kubernetes.io/projected/0a0da695-b68c-4390-abd9-9bd1383c5e87-kube-api-access-24fwq\") pod \"iptables-alerter-2bxkz\" (UID: \"0a0da695-b68c-4390-abd9-9bd1383c5e87\") " pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.454479 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.454461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zf4s\" (UniqueName: \"kubernetes.io/projected/58b72d20-daee-435f-a08b-856037109537-kube-api-access-9zf4s\") pod \"multus-additional-cni-plugins-hlw75\" (UID: \"58b72d20-daee-435f-a08b-856037109537\") " pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.455291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.455274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6hf\" (UniqueName: \"kubernetes.io/projected/379367f7-b885-4486-b103-e9d470325865-kube-api-access-cs6hf\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.463364 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.463329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"1421b2f48a6ec546abab4c3baa57eb62","Type":"ContainerStarted","Data":"23a76b722a5b6b87d2d496f07a29700fe03070a5b126a61b33bce5018a7d299d"}
Jul 29 01:10:14.497589 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.497566    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee83001d783d1446eea0c77a1a727ba.slice/crio-e2ed926e196fc70afb3a4faccb2cf99b91e2916e73b5189c37806b4397d655c0 WatchSource:0}: Error finding container e2ed926e196fc70afb3a4faccb2cf99b91e2916e73b5189c37806b4397d655c0: Status 404 returned error can't find the container with id e2ed926e196fc70afb3a4faccb2cf99b91e2916e73b5189c37806b4397d655c0
Jul 29 01:10:14.571130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.571062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfxjm"
Jul 29 01:10:14.578462 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.578442    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e9a927_2417_42cb_9e94_f42270a9d4e1.slice/crio-c580de4eb86475360db0c4741ed023a74114b225dfea52c09355b67177373519 WatchSource:0}: Error finding container c580de4eb86475360db0c4741ed023a74114b225dfea52c09355b67177373519: Status 404 returned error can't find the container with id c580de4eb86475360db0c4741ed023a74114b225dfea52c09355b67177373519
Jul 29 01:10:14.581577 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.581562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:14.589013 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.588992    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa339f5a_ecb3_4616_8e1e_02dba66601d8.slice/crio-22047d7c7d9744b9a1d971e440770c4e1a72db3c41f42819561d9b36ef5f6fb3 WatchSource:0}: Error finding container 22047d7c7d9744b9a1d971e440770c4e1a72db3c41f42819561d9b36ef5f6fb3: Status 404 returned error can't find the container with id 22047d7c7d9744b9a1d971e440770c4e1a72db3c41f42819561d9b36ef5f6fb3
Jul 29 01:10:14.601192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.601171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bxkz"
Jul 29 01:10:14.607839 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.607817    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0da695_b68c_4390_abd9_9bd1383c5e87.slice/crio-da59388c350df9980876152de31a3e5a4b5ab364d5b1a46c7f5ee16114c8d8fa WatchSource:0}: Error finding container da59388c350df9980876152de31a3e5a4b5ab364d5b1a46c7f5ee16114c8d8fa: Status 404 returned error can't find the container with id da59388c350df9980876152de31a3e5a4b5ab364d5b1a46c7f5ee16114c8d8fa
Jul 29 01:10:14.613910 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.613891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:14.622391 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.622372    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5ce9e2_f857_4a6b_af7a_2fb5f2333e27.slice/crio-6e588d2c24c1490d796488c08ef8bf949ba5cb6b173382acc77afd245868e1fa WatchSource:0}: Error finding container 6e588d2c24c1490d796488c08ef8bf949ba5cb6b173382acc77afd245868e1fa: Status 404 returned error can't find the container with id 6e588d2c24c1490d796488c08ef8bf949ba5cb6b173382acc77afd245868e1fa
Jul 29 01:10:14.630136 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.630102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc"
Jul 29 01:10:14.637358 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.637331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba7d09a_2198_469c_8971_19c969703358.slice/crio-dcc47b37f0ef1837a17325a7c87210115478f5ae9862ab1b8fbf2643d86c7a76 WatchSource:0}: Error finding container dcc47b37f0ef1837a17325a7c87210115478f5ae9862ab1b8fbf2643d86c7a76: Status 404 returned error can't find the container with id dcc47b37f0ef1837a17325a7c87210115478f5ae9862ab1b8fbf2643d86c7a76
Jul 29 01:10:14.651986 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.651960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mfgmj"
Jul 29 01:10:14.658666 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.658647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae9cb32_dcf4_4464_b83f_77ad30c84c7a.slice/crio-48f4cec5a8ad67751e78c056f90de0fa9fc2677ec93c52b948618f80f4e84580 WatchSource:0}: Error finding container 48f4cec5a8ad67751e78c056f90de0fa9fc2677ec93c52b948618f80f4e84580: Status 404 returned error can't find the container with id 48f4cec5a8ad67751e78c056f90de0fa9fc2677ec93c52b948618f80f4e84580
Jul 29 01:10:14.664655 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.664640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlw75"
Jul 29 01:10:14.671888 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.671868    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b72d20_daee_435f_a08b_856037109537.slice/crio-9ad25ac8f6633e2576346c3b58d77345332cf39544ced09619a42a084f501caf WatchSource:0}: Error finding container 9ad25ac8f6633e2576346c3b58d77345332cf39544ced09619a42a084f501caf: Status 404 returned error can't find the container with id 9ad25ac8f6633e2576346c3b58d77345332cf39544ced09619a42a084f501caf
Jul 29 01:10:14.682565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.682551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsfjt"
Jul 29 01:10:14.688327 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.688304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nz88"
Jul 29 01:10:14.689289 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.689271    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84ac7e9_e8a9_4357_aefc_48f48a65d883.slice/crio-7082b66ce4747beae79d6b3824f2972ca982c62878b96aef89b3390706b9d1c8 WatchSource:0}: Error finding container 7082b66ce4747beae79d6b3824f2972ca982c62878b96aef89b3390706b9d1c8: Status 404 returned error can't find the container with id 7082b66ce4747beae79d6b3824f2972ca982c62878b96aef89b3390706b9d1c8
Jul 29 01:10:14.695724 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:14.695704    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7f0007_c5de_4625_a970_c636ae3ab383.slice/crio-a590d3f01c9aae4d4fdd94f082239c9d08edc77f6d54f6a81757241a4bbcc8c9 WatchSource:0}: Error finding container a590d3f01c9aae4d4fdd94f082239c9d08edc77f6d54f6a81757241a4bbcc8c9: Status 404 returned error can't find the container with id a590d3f01c9aae4d4fdd94f082239c9d08edc77f6d54f6a81757241a4bbcc8c9
Jul 29 01:10:14.747757 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.747734    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:14.949671 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.949588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:14.949671 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:14.949637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:14.949884 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.949774    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:14.949884 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.949836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:15.949815345 +0000 UTC m=+3.103260708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:14.950282 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.950257    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:14.950368 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:14.950318    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:15.950300786 +0000 UTC m=+3.103746145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:15.051274 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.050615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:15.051274 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.050821    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:15.051274 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.050839    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:15.051274 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.050852    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:15.051274 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.050909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:16.050889676 +0000 UTC m=+3.204335047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:15.099878 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.099846    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:15.277369 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.277256    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 01:05:14 +0000 UTC" deadline="2028-02-11 00:38:06.078244029 +0000 UTC"
Jul 29 01:10:15.277369 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.277318    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h27m50.800930273s"
Jul 29 01:10:15.509859 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.509711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"6e588d2c24c1490d796488c08ef8bf949ba5cb6b173382acc77afd245868e1fa"}
Jul 29 01:10:15.532422 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.532334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bj86g" event={"ID":"fa339f5a-ecb3-4616-8e1e-02dba66601d8","Type":"ContainerStarted","Data":"22047d7c7d9744b9a1d971e440770c4e1a72db3c41f42819561d9b36ef5f6fb3"}
Jul 29 01:10:15.552763 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.552676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc" event={"ID":"bba7d09a-2198-469c-8971-19c969703358","Type":"ContainerStarted","Data":"dcc47b37f0ef1837a17325a7c87210115478f5ae9862ab1b8fbf2643d86c7a76"}
Jul 29 01:10:15.568196 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.568165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bxkz" event={"ID":"0a0da695-b68c-4390-abd9-9bd1383c5e87","Type":"ContainerStarted","Data":"da59388c350df9980876152de31a3e5a4b5ab364d5b1a46c7f5ee16114c8d8fa"}
Jul 29 01:10:15.594736 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.594016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfxjm" event={"ID":"47e9a927-2417-42cb-9e94-f42270a9d4e1","Type":"ContainerStarted","Data":"c580de4eb86475360db0c4741ed023a74114b225dfea52c09355b67177373519"}
Jul 29 01:10:15.609052 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.608891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" event={"ID":"1ee83001d783d1446eea0c77a1a727ba","Type":"ContainerStarted","Data":"e2ed926e196fc70afb3a4faccb2cf99b91e2916e73b5189c37806b4397d655c0"}
Jul 29 01:10:15.618759 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.618730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nz88" event={"ID":"0c7f0007-c5de-4625-a970-c636ae3ab383","Type":"ContainerStarted","Data":"a590d3f01c9aae4d4fdd94f082239c9d08edc77f6d54f6a81757241a4bbcc8c9"}
Jul 29 01:10:15.645879 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.645756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsfjt" event={"ID":"e84ac7e9-e8a9-4357-aefc-48f48a65d883","Type":"ContainerStarted","Data":"7082b66ce4747beae79d6b3824f2972ca982c62878b96aef89b3390706b9d1c8"}
Jul 29 01:10:15.668937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.664954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerStarted","Data":"9ad25ac8f6633e2576346c3b58d77345332cf39544ced09619a42a084f501caf"}
Jul 29 01:10:15.668937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.667921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfgmj" event={"ID":"bae9cb32-dcf4-4464-b83f-77ad30c84c7a","Type":"ContainerStarted","Data":"48f4cec5a8ad67751e78c056f90de0fa9fc2677ec93c52b948618f80f4e84580"}
Jul 29 01:10:15.932834 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.932761    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.959869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:15.959922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.960058    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.960145    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:17.960124688 +0000 UTC m=+5.113570065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.960543    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:15.960633 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:15.960593    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:17.960577313 +0000 UTC m=+5.114022673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:16.061001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.060965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:16.061195 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.061143    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:16.061195 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.061164    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:16.061195 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.061177    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:16.061396 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.061236    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:18.06121616 +0000 UTC m=+5.214661537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:16.238262 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.238176    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 01:10:16.278475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.278404    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 01:05:14 +0000 UTC" deadline="2028-04-05 06:01:40.339464412 +0000 UTC"
Jul 29 01:10:16.278475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.278466    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14788h51m24.061002773s"
Jul 29 01:10:16.460610 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.460575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:16.460818 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.460718    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:16.460818 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.460805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:16.460937 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.460878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:16.460995 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:16.460940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:16.461049 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:16.461023    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:17.978087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:17.978162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:17.978260    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:17.978286    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:17.978332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:21.978312235 +0000 UTC m=+9.131757593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:17.978365 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:17.978361    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:21.978350124 +0000 UTC m=+9.131795484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:18.079656 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:18.078949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:18.079656 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.079183    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:18.079656 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.079205    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:18.079656 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.079219    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:18.079656 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.079284    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:22.079263257 +0000 UTC m=+9.232708619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:18.460949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:18.460848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:18.460949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:18.460910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:18.461234 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.461046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:18.461423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:18.461397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:18.461517 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.461498    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:18.461727 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:18.461647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:20.461081 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:20.461032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:20.461527 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:20.461186    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:20.461597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:20.461573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:20.461722 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:20.461692    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:20.461839 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:20.461824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:20.461917 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:20.461900    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:22.011768 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.011729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:22.012182 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.011782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:22.012182 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.011886    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:22.012182 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.011934    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:30.011920449 +0000 UTC m=+17.165365803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:22.012182 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.011981    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:22.012182 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.012001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:30.011995146 +0000 UTC m=+17.165440500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:22.112174 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.112131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:22.112823 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.112366    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:22.112823 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.112422    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:22.112823 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.112437    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:22.112823 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.112497    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:30.112478694 +0000 UTC m=+17.265924062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:22.461444 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.461368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:22.461600 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.461469    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:22.461760 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.461742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:22.461836 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.461808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:22.461896 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:22.461850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:22.461896 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:22.461890    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:24.460722 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:24.460601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:24.460722 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:24.460674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:24.460722 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:24.460673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:24.461286 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:24.460777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:24.461286 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:24.460889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:24.461286 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:24.460970    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:26.460854 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:26.460821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:26.461341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:26.460909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:26.461341 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:26.460916    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:26.461341 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:26.460987    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:26.461341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:26.460996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:26.461341 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:26.461095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:28.460989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:28.460733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:28.461447 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:28.460759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:28.461447 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:28.461095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:28.461447 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:28.461172    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:28.461447 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:28.460733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:28.461447 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:28.461268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:30.073246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.073209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:30.073246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.073255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:30.073766 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.073350    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:30.073766 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.073415    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:46.073399289 +0000 UTC m=+33.226844643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:30.073766 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.073356    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:30.073766 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.073493    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:46.073476638 +0000 UTC m=+33.226921992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:30.173965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.173929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:30.174146 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.174105    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:30.174146 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.174124    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:30.174146 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.174135    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:30.174286 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.174216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:10:46.174196823 +0000 UTC m=+33.327642188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:30.461170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.461090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:30.461170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.461118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:30.461170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:30.461122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:30.461415 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.461215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:30.461415 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.461299    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:30.461415 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:30.461389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:32.460986 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:32.460954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:32.461481 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:32.461107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:32.461481 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:32.461114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:32.461481 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:32.461258    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:32.461481 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:32.461297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:32.461481 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:32.461383    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:34.460389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:34.460356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:34.460726 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:34.460463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:34.460726 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:34.460471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:34.460726 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:34.460493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:34.460726 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:34.460573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:34.460726 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:34.460635    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:35.723461 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"8de797d155a338939652be35033fd76452b9387d87985c76bbc4caeacab34ab2"}
Jul 29 01:10:35.724331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"f143a413078ae6dc6c0bcc1b5c6602ef3b6e91e7efa3376e1a0427b7de8748dd"}
Jul 29 01:10:35.724331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"a70a5fc6d2e9e73c7507da4330d7ebe88be99c2692db6f75ca418dd634ee8ec3"}
Jul 29 01:10:35.724331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"c2f5b7eb713195657de8cd6c30e85320ea51022cdc931f3cf2228dfcca07a5d8"}
Jul 29 01:10:35.724331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"39ae424c1cb5148c978e2e002f9b5d83bd622143457a6982f2c4f1146e37dec7"}
Jul 29 01:10:35.724331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.723701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"0944584bc7caaefbdd42f1ee5a09b206f55b1b2532ed207e5554878cd5a0b09a"}
Jul 29 01:10:35.724620 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.724590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bj86g" event={"ID":"fa339f5a-ecb3-4616-8e1e-02dba66601d8","Type":"ContainerStarted","Data":"c77d7ba65a1c9d7b9decca9ffa4cf3bc3fccc8394e72ba5d1667c4ea797d8ef9"}
Jul 29 01:10:35.726462 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.726428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc" event={"ID":"bba7d09a-2198-469c-8971-19c969703358","Type":"ContainerStarted","Data":"d289a655b6004a117827a3813d842a4b831f4c254407004c429f59fe4b311085"}
Jul 29 01:10:35.728158 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.728132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfxjm" event={"ID":"47e9a927-2417-42cb-9e94-f42270a9d4e1","Type":"ContainerStarted","Data":"1fcd17b26b1597843183aa4b88f19d44eb1f5eaca6cee6e9d1b0e032e1aaf5d4"}
Jul 29 01:10:35.729837 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.729807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" event={"ID":"1ee83001d783d1446eea0c77a1a727ba","Type":"ContainerStarted","Data":"76a3d5dcb1fdc90a10b2448bcf6840ad6d6ad5730bad6080530744befee17cb9"}
Jul 29 01:10:35.731385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.731362    2577 generic.go:356] "Generic (PLEG): container finished" podID="1421b2f48a6ec546abab4c3baa57eb62" containerID="2c3b01734df28b10460836e1435e29db961edec0dd12e53c72eea08eebf3bc77" exitCode=0
Jul 29 01:10:35.731485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.731448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"1421b2f48a6ec546abab4c3baa57eb62","Type":"ContainerDied","Data":"2c3b01734df28b10460836e1435e29db961edec0dd12e53c72eea08eebf3bc77"}
Jul 29 01:10:35.733307 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.733286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nz88" event={"ID":"0c7f0007-c5de-4625-a970-c636ae3ab383","Type":"ContainerStarted","Data":"e3af80ff852c752aaf616d64f235557ebf69a712a2cc005cc2c9324566d27b33"}
Jul 29 01:10:35.734734 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.734710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsfjt" event={"ID":"e84ac7e9-e8a9-4357-aefc-48f48a65d883","Type":"ContainerStarted","Data":"8c6f177b38ddf684480a45008f4d2e02d0acea5426e0d572369b9a200c7484f9"}
Jul 29 01:10:35.736236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.736213    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="574ea4f214ce1b45b48f72eced19a89584afc2fba0d3d66810e144430c6b5fad" exitCode=0
Jul 29 01:10:35.736340 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.736290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"574ea4f214ce1b45b48f72eced19a89584afc2fba0d3d66810e144430c6b5fad"}
Jul 29 01:10:35.737911 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.737888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mfgmj" event={"ID":"bae9cb32-dcf4-4464-b83f-77ad30c84c7a","Type":"ContainerStarted","Data":"d375e3119bf6b267bf1dff8fab690cdc9255a85872331f1d03b51668e8baf2df"}
Jul 29 01:10:35.747078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.747021    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bj86g" podStartSLOduration=2.5693959079999997 podStartE2EDuration="22.747009387s" totalImagesPullingTime="20.177613479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.746196252 +0000 UTC m=+22.899641629" watchObservedRunningTime="2026-07-29 01:10:35.747009387 +0000 UTC m=+22.900454763"
Jul 29 01:10:35.769330 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.769280    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xsfjt" podStartSLOduration=2.670091723 podStartE2EDuration="22.769263638s" totalImagesPullingTime="20.099171915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.769102228 +0000 UTC m=+22.922547604" watchObservedRunningTime="2026-07-29 01:10:35.769263638 +0000 UTC m=+22.922709016"
Jul 29 01:10:35.790455 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.786656    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" podStartSLOduration=21.786638792 podStartE2EDuration="21.786638792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.785621917 +0000 UTC m=+22.939067294" watchObservedRunningTime="2026-07-29 01:10:35.786638792 +0000 UTC m=+22.940084166"
Jul 29 01:10:35.807943 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.807845    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hfxjm" podStartSLOduration=2.595317166 podStartE2EDuration="22.807830843s" totalImagesPullingTime="20.212513677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.807385896 +0000 UTC m=+22.960831273" watchObservedRunningTime="2026-07-29 01:10:35.807830843 +0000 UTC m=+22.961276218"
Jul 29 01:10:35.826166 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.826133    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mfgmj" podStartSLOduration=2.708680356 podStartE2EDuration="22.826119585s" totalImagesPullingTime="20.117439229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.825759233 +0000 UTC m=+22.979204608" watchObservedRunningTime="2026-07-29 01:10:35.826119585 +0000 UTC m=+22.979564960"
Jul 29 01:10:35.869859 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:35.869821    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9nz88" podStartSLOduration=2.570228328 podStartE2EDuration="22.869808086s" totalImagesPullingTime="20.299579758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:35.869295581 +0000 UTC m=+23.022740959" watchObservedRunningTime="2026-07-29 01:10:35.869808086 +0000 UTC m=+23.023253463"
Jul 29 01:10:36.460900 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.460872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:36.461080 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.460877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:36.461080 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:36.461011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:36.461080 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.460877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:36.461316 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:36.461095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:36.461316 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:36.461157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:36.517549 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.517503    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 01:10:36.742625 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.742592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"1421b2f48a6ec546abab4c3baa57eb62","Type":"ContainerStarted","Data":"e16002bcdfd30b0585af1ab7f1c87b046e97e0ebcafdf7cf417fa0b1dde1c633"}
Jul 29 01:10:36.744694 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.744671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc" event={"ID":"bba7d09a-2198-469c-8971-19c969703358","Type":"ContainerStarted","Data":"b4d61c80a84f3403d0e44e6ab899aa9636143cb05f618cc844aa82b6f52e2180"}
Jul 29 01:10:36.746619 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.746392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bxkz" event={"ID":"0a0da695-b68c-4390-abd9-9bd1383c5e87","Type":"ContainerStarted","Data":"f6769ece990fb1a23d7907ba175f68855ab43ee43e0ffed31eb7c026bd6acefb"}
Jul 29 01:10:36.757496 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.757417    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" podStartSLOduration=22.757402381 podStartE2EDuration="22.757402381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:36.75728284 +0000 UTC m=+23.910728215" watchObservedRunningTime="2026-07-29 01:10:36.757402381 +0000 UTC m=+23.910847758"
Jul 29 01:10:36.770531 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:36.770493    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bxkz" podStartSLOduration=3.611555532 podStartE2EDuration="23.770481089s" totalImagesPullingTime="20.158925557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:36.770389498 +0000 UTC m=+23.923834873" watchObservedRunningTime="2026-07-29 01:10:36.770481089 +0000 UTC m=+23.923926499"
Jul 29 01:10:37.302964 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.302833    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T01:10:36.517530332Z","UUID":"d31bbc51-0fee-4c36-b85e-03cd5c1ef296","Handler":null,"Name":"","Endpoint":""}
Jul 29 01:10:37.304494 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.304471    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 01:10:37.304624 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.304520    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 01:10:37.646143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.646112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:37.646801 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.646777    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:37.755152 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.755050    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:37.755994 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:37.755978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bj86g"
Jul 29 01:10:38.460663 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.460635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:38.460849 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.460671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:38.460849 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.460738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:38.460849 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:38.460753    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:38.460849 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:38.460808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:38.461097 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:38.460912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:38.753220 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.753120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc" event={"ID":"bba7d09a-2198-469c-8971-19c969703358","Type":"ContainerStarted","Data":"3addf0918dd2a8a02e86fa9f11637f327b7b7cb9e9069721aebf29f87d1675ba"}
Jul 29 01:10:38.756914 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.756888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"5e6d9ff7225dcd91deb14ea0f419c4a9f45d3913c2fef4c7dbcc5d5b6ccac48b"}
Jul 29 01:10:38.773372 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:38.773333    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdtmc" podStartSLOduration=3.3340058360000002 podStartE2EDuration="25.77331788s" totalImagesPullingTime="22.439312044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:38.773031104 +0000 UTC m=+25.926476481" watchObservedRunningTime="2026-07-29 01:10:38.77331788 +0000 UTC m=+25.926763261"
Jul 29 01:10:40.460545 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.460305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:40.461259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.460325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:40.461259 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:40.460573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:40.461259 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:40.460644    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:40.461259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.460305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:40.461259 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:40.460773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:40.763149 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.763078    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="56c9c1634a8f1a69536efd340cff72d629b6ed984527ec682c26c2c103a26036" exitCode=0
Jul 29 01:10:40.763332 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.763175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"56c9c1634a8f1a69536efd340cff72d629b6ed984527ec682c26c2c103a26036"}
Jul 29 01:10:40.766837 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.766816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" event={"ID":"7d5ce9e2-f857-4a6b-af7a-2fb5f2333e27","Type":"ContainerStarted","Data":"c0a19ced509e6925dd86670d36976df86a99c7dbea877e738a6f2579d2490ea3"}
Jul 29 01:10:40.767086 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.767069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:40.767138 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.767126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:40.767190 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.767180    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:40.782278 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.782259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:40.782407 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.782386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:10:40.811424 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:40.811386    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb" podStartSLOduration=7.6156609 podStartE2EDuration="27.811373654s" totalImagesPullingTime="20.195712754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:40.809868734 +0000 UTC m=+27.963314110" watchObservedRunningTime="2026-07-29 01:10:40.811373654 +0000 UTC m=+27.964819031"
Jul 29 01:10:42.195714 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.195438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdhzb"]
Jul 29 01:10:42.196151 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.195731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:42.196151 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:42.195834    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:42.198282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.198257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctvcr"]
Jul 29 01:10:42.198402 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.198348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:42.198452 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:42.198432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:42.199016 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.198994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsrgh"]
Jul 29 01:10:42.199132 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.199089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:42.199218 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:42.199189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:42.773479 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.773443    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="b5bad93fcab367dd57966a6138aaa83fd39d9c669b08749c03f185225fef7545" exitCode=0
Jul 29 01:10:42.773668 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:42.773526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"b5bad93fcab367dd57966a6138aaa83fd39d9c669b08749c03f185225fef7545"}
Jul 29 01:10:43.461911 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:43.461879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:43.462315 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:43.461971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:43.462315 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:43.462002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:43.462315 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:43.462068    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:44.460322 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:44.460291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:44.460468 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:44.460394    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:44.779340 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:44.779271    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="7f359b0b12022f1d01630c022685ec6af6cadd2954ec7cb4a7f20f3d36bfead1" exitCode=0
Jul 29 01:10:44.779340 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:44.779328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"7f359b0b12022f1d01630c022685ec6af6cadd2954ec7cb4a7f20f3d36bfead1"}
Jul 29 01:10:45.460862 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:45.460675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:45.461014 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:45.460945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:45.461103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:45.461076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:45.461197 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:45.461177    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:46.096235 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:46.096195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:46.096640 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:46.096236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:46.096640 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.096359    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:46.096640 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.096373    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:46.096640 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.096408    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:11:18.096394777 +0000 UTC m=+65.249840134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 01:10:46.096640 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.096434    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret podName:d5adf746-cbd1-47d6-a225-636fdbc225c4 nodeName:}" failed. No retries permitted until 2026-07-29 01:11:18.096415452 +0000 UTC m=+65.249860807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret") pod "global-pull-secret-syncer-qdhzb" (UID: "d5adf746-cbd1-47d6-a225-636fdbc225c4") : object "kube-system"/"original-pull-secret" not registered
Jul 29 01:10:46.196919 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:46.196876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:46.197102 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.197004    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 01:10:46.197102 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.197026    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 01:10:46.197102 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.197053    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cjf55 for pod openshift-network-diagnostics/network-check-target-hsrgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:46.197242 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.197107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55 podName:66c610bf-aaf3-4b1d-8c75-22e3e6404111 nodeName:}" failed. No retries permitted until 2026-07-29 01:11:18.197093736 +0000 UTC m=+65.350539109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjf55" (UniqueName: "kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55") pod "network-check-target-hsrgh" (UID: "66c610bf-aaf3-4b1d-8c75-22e3e6404111") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 01:10:46.461054 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:46.461009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:46.461231 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:46.461131    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:10:47.460977 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.460943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:47.461377 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:47.461093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdhzb" podUID="d5adf746-cbd1-47d6-a225-636fdbc225c4"
Jul 29 01:10:47.461377 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.461145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:47.461377 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:47.461232    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsrgh" podUID="66c610bf-aaf3-4b1d-8c75-22e3e6404111"
Jul 29 01:10:47.666903 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.666874    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeReady"
Jul 29 01:10:47.667078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.667025    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 01:10:47.717260 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.717191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xnxd6"]
Jul 29 01:10:47.720305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.720283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.722796 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.722777    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jq8v5"]
Jul 29 01:10:47.722911 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.722879    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 01:10:47.723108 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.723026    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 01:10:47.723108 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.723090    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ccxn7\""
Jul 29 01:10:47.723264 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.723248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 01:10:47.724955 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.724936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.728529 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.727350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 01:10:47.728529 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.727385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 01:10:47.728529 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.727479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rpdf\""
Jul 29 01:10:47.730385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.730364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnxd6"]
Jul 29 01:10:47.737467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.737331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jq8v5"]
Jul 29 01:10:47.812100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-cert\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.812279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e652714-955b-4d9d-a768-f781cf121cea-tmp-dir\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.812279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zn4\" (UniqueName: \"kubernetes.io/projected/1e652714-955b-4d9d-a768-f781cf121cea-kube-api-access-g8zn4\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.812279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7776z\" (UniqueName: \"kubernetes.io/projected/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-kube-api-access-7776z\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.812433 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e652714-955b-4d9d-a768-f781cf121cea-config-volume\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.812433 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.812334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.913365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e652714-955b-4d9d-a768-f781cf121cea-config-volume\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.913365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-cert\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:47.913477    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e652714-955b-4d9d-a768-f781cf121cea-tmp-dir\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:47.913526    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:10:48.413510894 +0000 UTC m=+35.566956248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zn4\" (UniqueName: \"kubernetes.io/projected/1e652714-955b-4d9d-a768-f781cf121cea-kube-api-access-g8zn4\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.913594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7776z\" (UniqueName: \"kubernetes.io/projected/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-kube-api-access-7776z\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.913905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.913801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e652714-955b-4d9d-a768-f781cf121cea-tmp-dir\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.914293 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.914273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e652714-955b-4d9d-a768-f781cf121cea-config-volume\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:47.918348 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.918325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-cert\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.922806 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.922769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7776z\" (UniqueName: \"kubernetes.io/projected/1fd6be0d-2df3-4faf-9974-ca1d237b2c33-kube-api-access-7776z\") pod \"ingress-canary-xnxd6\" (UID: \"1fd6be0d-2df3-4faf-9974-ca1d237b2c33\") " pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:47.923147 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:47.923108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zn4\" (UniqueName: \"kubernetes.io/projected/1e652714-955b-4d9d-a768-f781cf121cea-kube-api-access-g8zn4\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:48.036456 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.036385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnxd6"
Jul 29 01:10:48.417430 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.417395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:48.417598 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:48.417567    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:10:48.417668 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:48.417654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:10:49.417630853 +0000 UTC m=+36.571076206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:10:48.460681 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.460641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:10:48.463248 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.463219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 01:10:48.463838 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.463301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-llbnf\""
Jul 29 01:10:48.930224 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:48.930076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnxd6"]
Jul 29 01:10:48.933378 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:10:48.933337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6be0d_2df3_4faf_9974_ca1d237b2c33.slice/crio-2149f4a9f890a86d52b902d77f2cdb4f5deb0e75d374067b474d084c1b027001 WatchSource:0}: Error finding container 2149f4a9f890a86d52b902d77f2cdb4f5deb0e75d374067b474d084c1b027001: Status 404 returned error can't find the container with id 2149f4a9f890a86d52b902d77f2cdb4f5deb0e75d374067b474d084c1b027001
Jul 29 01:10:49.425421 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.425386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:49.425558 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:49.425501    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:10:49.425558 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:49.425552    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:10:51.425539026 +0000 UTC m=+38.578984381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:10:49.461007 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.460981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:10:49.461215 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.461185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:10:49.463517 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.463500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 01:10:49.463517 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.463509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 01:10:49.464487 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.464471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d72dr\""
Jul 29 01:10:49.464566 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.464498    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 01:10:49.795026 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.794941    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="36e5c6fada4bf3e3197cf36f8173bd4ffc85e161fd50b7f7b5f2aa9816f5b16b" exitCode=0
Jul 29 01:10:49.795208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.795029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"36e5c6fada4bf3e3197cf36f8173bd4ffc85e161fd50b7f7b5f2aa9816f5b16b"}
Jul 29 01:10:49.796350 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:49.796308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnxd6" event={"ID":"1fd6be0d-2df3-4faf-9974-ca1d237b2c33","Type":"ContainerStarted","Data":"2149f4a9f890a86d52b902d77f2cdb4f5deb0e75d374067b474d084c1b027001"}
Jul 29 01:10:50.802555 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:50.802523    2577 generic.go:356] "Generic (PLEG): container finished" podID="58b72d20-daee-435f-a08b-856037109537" containerID="28c53c09922421a0a35e90a1d5044a21e8299fe4eaf580a2c3bf89e00e9249cc" exitCode=0
Jul 29 01:10:50.802982 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:50.802575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerDied","Data":"28c53c09922421a0a35e90a1d5044a21e8299fe4eaf580a2c3bf89e00e9249cc"}
Jul 29 01:10:51.440910 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:51.440832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:51.441074 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:51.440959    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:10:51.441074 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:51.441013    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:10:55.441000032 +0000 UTC m=+42.594445385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:10:51.805996 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:51.805919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnxd6" event={"ID":"1fd6be0d-2df3-4faf-9974-ca1d237b2c33","Type":"ContainerStarted","Data":"e863464771a5603e3b359a07a43c340f06e5e44b187992fd1249a768114f801f"}
Jul 29 01:10:51.809235 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:51.809206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlw75" event={"ID":"58b72d20-daee-435f-a08b-856037109537","Type":"ContainerStarted","Data":"7f0f4583f3d86548e8774d52aa51b1339b8e1a71cd27c4eecff060acf7854a27"}
Jul 29 01:10:51.821943 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:51.821905    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xnxd6" podStartSLOduration=2.523330519 podStartE2EDuration="4.821893157s" totalImagesPullingTime="2.298562638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:51.820725864 +0000 UTC m=+38.974171241" watchObservedRunningTime="2026-07-29 01:10:51.821893157 +0000 UTC m=+38.975338511"
Jul 29 01:10:51.841862 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:51.841823    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hlw75" podStartSLOduration=8.256257872 podStartE2EDuration="38.841805271s" totalImagesPullingTime="30.585547399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:10:51.840359038 +0000 UTC m=+38.993804413" watchObservedRunningTime="2026-07-29 01:10:51.841805271 +0000 UTC m=+38.995250646"
Jul 29 01:10:55.468823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:10:55.468614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:10:55.469169 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:55.468758    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:10:55.469169 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:10:55.468914    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:11:03.468898443 +0000 UTC m=+50.622343798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:11:03.523837 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:03.523798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:11:03.524350 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:03.523936    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:11:03.524350 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:03.523994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:11:19.523979308 +0000 UTC m=+66.677424662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:11:12.793427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:12.793397    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dfrwb"
Jul 29 01:11:18.125486 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.125443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:11:18.125486 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.125485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:11:18.128580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.128558    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 01:11:18.128734 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.128715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 01:11:18.136136 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:18.136112    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 01:11:18.136222 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:18.136188    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:12:22.136171837 +0000 UTC m=+129.289617191 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : secret "metrics-daemon-secret" not found
Jul 29 01:11:18.139714 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.139696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d5adf746-cbd1-47d6-a225-636fdbc225c4-original-pull-secret\") pod \"global-pull-secret-syncer-qdhzb\" (UID: \"d5adf746-cbd1-47d6-a225-636fdbc225c4\") " pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:11:18.225908 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.225854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:11:18.228591 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.228573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 01:11:18.239515 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.239493    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 01:11:18.249889 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.249862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjf55\" (UniqueName: \"kubernetes.io/projected/66c610bf-aaf3-4b1d-8c75-22e3e6404111-kube-api-access-cjf55\") pod \"network-check-target-hsrgh\" (UID: \"66c610bf-aaf3-4b1d-8c75-22e3e6404111\") " pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:11:18.273502 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.273476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d72dr\""
Jul 29 01:11:18.275503 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.275488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdhzb"
Jul 29 01:11:18.282451 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.282426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:11:18.406715 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.406684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdhzb"]
Jul 29 01:11:18.409896 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:11:18.409871    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5adf746_cbd1_47d6_a225_636fdbc225c4.slice/crio-aad3d8e19ad95a4f8b62c8157b368e47b53f3f8fa1a328b65041c83e889b4478 WatchSource:0}: Error finding container aad3d8e19ad95a4f8b62c8157b368e47b53f3f8fa1a328b65041c83e889b4478: Status 404 returned error can't find the container with id aad3d8e19ad95a4f8b62c8157b368e47b53f3f8fa1a328b65041c83e889b4478
Jul 29 01:11:18.423007 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.422980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsrgh"]
Jul 29 01:11:18.427006 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:11:18.426973    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c610bf_aaf3_4b1d_8c75_22e3e6404111.slice/crio-400ea27918bd887b33fc4e5b55052d5c70d53be53706957e7d6e14fbe957795b WatchSource:0}: Error finding container 400ea27918bd887b33fc4e5b55052d5c70d53be53706957e7d6e14fbe957795b: Status 404 returned error can't find the container with id 400ea27918bd887b33fc4e5b55052d5c70d53be53706957e7d6e14fbe957795b
Jul 29 01:11:18.871208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.871171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsrgh" event={"ID":"66c610bf-aaf3-4b1d-8c75-22e3e6404111","Type":"ContainerStarted","Data":"400ea27918bd887b33fc4e5b55052d5c70d53be53706957e7d6e14fbe957795b"}
Jul 29 01:11:18.872368 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:18.872329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdhzb" event={"ID":"d5adf746-cbd1-47d6-a225-636fdbc225c4","Type":"ContainerStarted","Data":"aad3d8e19ad95a4f8b62c8157b368e47b53f3f8fa1a328b65041c83e889b4478"}
Jul 29 01:11:19.534794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:19.534753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:11:19.535328 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:19.534892    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:11:19.535328 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:19.534984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:11:51.534962341 +0000 UTC m=+98.688407714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:11:22.885948 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:22.885258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsrgh" event={"ID":"66c610bf-aaf3-4b1d-8c75-22e3e6404111","Type":"ContainerStarted","Data":"9c612c30921370aa1648c9f6880f5d83d4c07d00d0a89b2171b4ec32a35341a1"}
Jul 29 01:11:22.886380 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:22.885963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:11:22.887558 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:22.887531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdhzb" event={"ID":"d5adf746-cbd1-47d6-a225-636fdbc225c4","Type":"ContainerStarted","Data":"0077051a8497428e90a614e2163ffe3278a18138b136164174cd9d8cbec832f5"}
Jul 29 01:11:22.901721 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:22.901671    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hsrgh" podStartSLOduration=65.601094447 podStartE2EDuration="1m9.901655193s" totalImagesPullingTime="4.300560746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:11:22.900882589 +0000 UTC m=+70.054327964" watchObservedRunningTime="2026-07-29 01:11:22.901655193 +0000 UTC m=+70.055100570"
Jul 29 01:11:22.916034 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:22.915991    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qdhzb" podStartSLOduration=65.594584036 podStartE2EDuration="1m9.915978878s" totalImagesPullingTime="4.321394842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:11:22.915558164 +0000 UTC m=+70.069003541" watchObservedRunningTime="2026-07-29 01:11:22.915978878 +0000 UTC m=+70.069424253"
Jul 29 01:11:51.549094 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:51.549057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:11:51.549627 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:51.549148    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:11:51.549627 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:11:51.549222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:12:55.549201709 +0000 UTC m=+162.702647072 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:11:54.895292 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:11:54.895266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hsrgh"
Jul 29 01:12:22.147824 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:22.147778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:12:22.148242 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:22.147920    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 01:12:22.148242 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:22.147981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:24.147967124 +0000 UTC m=+251.301412478 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : secret "metrics-daemon-secret" not found
Jul 29 01:12:38.059896 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.059864    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"]
Jul 29 01:12:38.062536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.062519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.066639 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.066613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 01:12:38.066739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.066641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 01:12:38.066739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.066717    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fgpqn\""
Jul 29 01:12:38.066739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.066731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 01:12:38.067417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.067398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 01:12:38.074467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.074446    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"]
Jul 29 01:12:38.077167 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.077151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"]
Jul 29 01:12:38.077253 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.077222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.079701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.079681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 01:12:38.079798 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.079747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 01:12:38.079869 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.079857    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 01:12:38.080102 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.080086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 01:12:38.086931 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.086911    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"]
Jul 29 01:12:38.149053 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/285c909f-e942-4d3c-bb8e-62c5f98a4b09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.149053 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.149207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fabdd6-e856-437b-99ca-061b36ceb0fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.149207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2dq\" (UniqueName: \"kubernetes.io/projected/c4fabdd6-e856-437b-99ca-061b36ceb0fc-kube-api-access-6k2dq\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.149207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsbh\" (UniqueName: \"kubernetes.io/projected/285c909f-e942-4d3c-bb8e-62c5f98a4b09-kube-api-access-9jsbh\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.149207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.149207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.149379 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.149226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249713 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsbh\" (UniqueName: \"kubernetes.io/projected/285c909f-e942-4d3c-bb8e-62c5f98a4b09-kube-api-access-9jsbh\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.249782 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249782 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249782 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/285c909f-e942-4d3c-bb8e-62c5f98a4b09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.249905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fabdd6-e856-437b-99ca-061b36ceb0fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.249905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.249895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2dq\" (UniqueName: \"kubernetes.io/projected/c4fabdd6-e856-437b-99ca-061b36ceb0fc-kube-api-access-6k2dq\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.250699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.250673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fabdd6-e856-437b-99ca-061b36ceb0fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.252354 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.252332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.252444 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.252423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/285c909f-e942-4d3c-bb8e-62c5f98a4b09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.252586 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.252567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.252622 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.252582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.252765 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.252751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c4fabdd6-e856-437b-99ca-061b36ceb0fc-ca\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.258614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.258589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsbh\" (UniqueName: \"kubernetes.io/projected/285c909f-e942-4d3c-bb8e-62c5f98a4b09-kube-api-access-9jsbh\") pod \"managed-serviceaccount-addon-agent-bbc5f6dd4-pc627\" (UID: \"285c909f-e942-4d3c-bb8e-62c5f98a4b09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.258966 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.258946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2dq\" (UniqueName: \"kubernetes.io/projected/c4fabdd6-e856-437b-99ca-061b36ceb0fc-kube-api-access-6k2dq\") pod \"cluster-proxy-proxy-agent-7d6d6fd84-hnkn7\" (UID: \"c4fabdd6-e856-437b-99ca-061b36ceb0fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.383512 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.383492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"
Jul 29 01:12:38.390229 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.390205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:12:38.506964 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.506937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627"]
Jul 29 01:12:38.511326 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:12:38.511302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod285c909f_e942_4d3c_bb8e_62c5f98a4b09.slice/crio-334bd931e4e59f7f7778613d3cf6e150e9807fdafe4bd4874db0455ceacbeb60 WatchSource:0}: Error finding container 334bd931e4e59f7f7778613d3cf6e150e9807fdafe4bd4874db0455ceacbeb60: Status 404 returned error can't find the container with id 334bd931e4e59f7f7778613d3cf6e150e9807fdafe4bd4874db0455ceacbeb60
Jul 29 01:12:38.525858 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:38.525837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"]
Jul 29 01:12:38.528797 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:12:38.528776    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4fabdd6_e856_437b_99ca_061b36ceb0fc.slice/crio-acd74654e3f31ec3cdf9fd04f5baf37e1352cfcbe74e7ab234863817842bbfa3 WatchSource:0}: Error finding container acd74654e3f31ec3cdf9fd04f5baf37e1352cfcbe74e7ab234863817842bbfa3: Status 404 returned error can't find the container with id acd74654e3f31ec3cdf9fd04f5baf37e1352cfcbe74e7ab234863817842bbfa3
Jul 29 01:12:39.069533 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:39.069499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627" event={"ID":"285c909f-e942-4d3c-bb8e-62c5f98a4b09","Type":"ContainerStarted","Data":"334bd931e4e59f7f7778613d3cf6e150e9807fdafe4bd4874db0455ceacbeb60"}
Jul 29 01:12:39.070573 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:39.070550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerStarted","Data":"acd74654e3f31ec3cdf9fd04f5baf37e1352cfcbe74e7ab234863817842bbfa3"}
Jul 29 01:12:42.082701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:42.082669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627" event={"ID":"285c909f-e942-4d3c-bb8e-62c5f98a4b09","Type":"ContainerStarted","Data":"c0325f480cca5f210bc62afea0c6cb3eb817cc86d6a93d179a87ffded7f044b8"}
Jul 29 01:12:42.098951 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:42.098903    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627" podStartSLOduration=1.241559999 podStartE2EDuration="4.098886599s" totalImagesPullingTime="2.8573266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:42.097345416 +0000 UTC m=+149.250790794" watchObservedRunningTime="2026-07-29 01:12:42.098886599 +0000 UTC m=+149.252332119"
Jul 29 01:12:43.087632 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:43.087599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerStarted","Data":"63d2b8bf031dabb9c412419d3df17fb133db5c80bacb52f05f27e9c3a8789776"}
Jul 29 01:12:43.087959 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:43.087641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerStarted","Data":"a1de6139cd43a59a2ab099d246f01e87c2a30c2f145d588c4786770731c202f5"}
Jul 29 01:12:43.087959 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:43.087653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerStarted","Data":"7ac277122b8106429f0b34db05cca6bc3e90de67992ccbd3fcdab1a762b040d3"}
Jul 29 01:12:43.108499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:43.108455    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" podStartSLOduration=0.904279847 podStartE2EDuration="5.10844066s" totalImagesPullingTime="4.204160813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:12:43.107824478 +0000 UTC m=+150.261269854" watchObservedRunningTime="2026-07-29 01:12:43.10844066 +0000 UTC m=+150.261886036"
Jul 29 01:12:50.743033 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:50.742991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jq8v5" podUID="1e652714-955b-4d9d-a768-f781cf121cea"
Jul 29 01:12:51.112964 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:51.112935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:12:51.471224 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:51.471142    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:12:55.567312 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:12:55.567281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:12:55.567673 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:55.567378    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 01:12:55.567673 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:12:55.567432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls podName:1e652714-955b-4d9d-a768-f781cf121cea nodeName:}" failed. No retries permitted until 2026-07-29 01:14:57.567416012 +0000 UTC m=+284.720861381 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls") pod "dns-default-jq8v5" (UID: "1e652714-955b-4d9d-a768-f781cf121cea") : secret "dns-default-metrics-tls" not found
Jul 29 01:13:05.461485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:13:05.461406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:14:24.238331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:24.238296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:14:24.238797 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:24.238407    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 01:14:24.238797 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:24.238471    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs podName:379367f7-b885-4486-b103-e9d470325865 nodeName:}" failed. No retries permitted until 2026-07-29 01:16:26.238455965 +0000 UTC m=+373.391901318 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs") pod "network-metrics-daemon-ctvcr" (UID: "379367f7-b885-4486-b103-e9d470325865") : secret "metrics-daemon-secret" not found
Jul 29 01:14:36.274214 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.274140    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2z2kk"]
Jul 29 01:14:36.276878 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.276860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.279923 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.279896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 01:14:36.280645 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.280628    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 01:14:36.280704 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.280663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2fjq\""
Jul 29 01:14:36.280704 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.280677    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 01:14:36.281367 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.281351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 01:14:36.289932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.289911    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2z2kk"]
Jul 29 01:14:36.415431 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.415408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8fc14f1-4a24-4d26-94d4-4c814c404f22-data-volume\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.415538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.415444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.415538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.415461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkf9v\" (UniqueName: \"kubernetes.io/projected/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-api-access-gkf9v\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.415621 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.415535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.415621 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.415574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc14f1-4a24-4d26-94d4-4c814c404f22-crio-socket\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516464 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516535 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc14f1-4a24-4d26-94d4-4c814c404f22-crio-socket\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516535 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8fc14f1-4a24-4d26-94d4-4c814c404f22-data-volume\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516607 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516607 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9v\" (UniqueName: \"kubernetes.io/projected/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-api-access-gkf9v\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516607 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc14f1-4a24-4d26-94d4-4c814c404f22-crio-socket\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.516607 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:36.516584    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:36.516743 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:36.516615    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 01:14:36.516743 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:36.516650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:37.016630326 +0000 UTC m=+264.170075701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:36.516743 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:36.516676    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:37.01665725 +0000 UTC m=+264.170102604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : configmap "kube-rbac-proxy" not found
Jul 29 01:14:36.516877 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.516820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8fc14f1-4a24-4d26-94d4-4c814c404f22-data-volume\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:36.527424 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:36.527368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf9v\" (UniqueName: \"kubernetes.io/projected/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-api-access-gkf9v\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:37.020023 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:37.019994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:37.020207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:37.020034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:37.020207 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:37.020132    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 01:14:37.020207 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:37.020145    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:37.020367 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:37.020216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:38.020198441 +0000 UTC m=+265.173643798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:37.020367 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:37.020232    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:38.02022518 +0000 UTC m=+265.173670534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : configmap "kube-rbac-proxy" not found
Jul 29 01:14:38.027856 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:38.027814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:38.028299 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:38.027903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:38.028299 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:38.027971    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:38.028299 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:38.027982    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 01:14:38.028299 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:38.028034    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:40.028017525 +0000 UTC m=+267.181462881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:38.028299 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:38.028065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:40.028057549 +0000 UTC m=+267.181502902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : configmap "kube-rbac-proxy" not found
Jul 29 01:14:38.453925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:38.453894    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsfjt_e84ac7e9-e8a9-4357-aefc-48f48a65d883/dns-node-resolver/0.log"
Jul 29 01:14:39.253634 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:39.253608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfgmj_bae9cb32-dcf4-4464-b83f-77ad30c84c7a/node-ca/0.log"
Jul 29 01:14:40.043093 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:40.043055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:40.043283 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:40.043145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:40.043283 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:40.043220    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:40.043401 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:40.043287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:44.043272217 +0000 UTC m=+271.196717572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:40.043401 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:40.043227    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 01:14:40.043401 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:40.043360    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:44.043343343 +0000 UTC m=+271.196788714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : configmap "kube-rbac-proxy" not found
Jul 29 01:14:40.254574 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:40.254547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xnxd6_1fd6be0d-2df3-4faf-9974-ca1d237b2c33/serve-healthcheck-canary/0.log"
Jul 29 01:14:42.421912 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:42.421883    2577 generic.go:356] "Generic (PLEG): container finished" podID="285c909f-e942-4d3c-bb8e-62c5f98a4b09" containerID="c0325f480cca5f210bc62afea0c6cb3eb817cc86d6a93d179a87ffded7f044b8" exitCode=255
Jul 29 01:14:42.422305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:42.421928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627" event={"ID":"285c909f-e942-4d3c-bb8e-62c5f98a4b09","Type":"ContainerDied","Data":"c0325f480cca5f210bc62afea0c6cb3eb817cc86d6a93d179a87ffded7f044b8"}
Jul 29 01:14:42.422305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:42.422217    2577 scope.go:117] "RemoveContainer" containerID="c0325f480cca5f210bc62afea0c6cb3eb817cc86d6a93d179a87ffded7f044b8"
Jul 29 01:14:43.426374 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:43.426341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bbc5f6dd4-pc627" event={"ID":"285c909f-e942-4d3c-bb8e-62c5f98a4b09","Type":"ContainerStarted","Data":"a00e982686d10197b04d5b1a69221eed74a96a17e435e1813146397cd52e9299"}
Jul 29 01:14:44.070508 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:44.070462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:44.070681 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:44.070545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:44.070681 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:44.070611    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 01:14:44.070681 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:44.070620    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:44.070681 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:44.070669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:52.070652473 +0000 UTC m=+279.224097828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : configmap "kube-rbac-proxy" not found
Jul 29 01:14:44.070681 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:44.070682    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:14:52.070676888 +0000 UTC m=+279.224122241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:52.127395 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:52.127363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:52.127786 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:52.127435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:52.127786 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:52.127466    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:52.127786 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:52.127525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls podName:c8fc14f1-4a24-4d26-94d4-4c814c404f22 nodeName:}" failed. No retries permitted until 2026-07-29 01:15:08.127509688 +0000 UTC m=+295.280955042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2z2kk" (UID: "c8fc14f1-4a24-4d26-94d4-4c814c404f22") : secret "insights-runtime-extractor-tls" not found
Jul 29 01:14:52.127894 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:52.127827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8fc14f1-4a24-4d26-94d4-4c814c404f22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:14:54.114297 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:14:54.114248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jq8v5" podUID="1e652714-955b-4d9d-a768-f781cf121cea"
Jul 29 01:14:54.455171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:54.455103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:14:57.664949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:57.664912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:14:57.667089 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:57.667070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e652714-955b-4d9d-a768-f781cf121cea-metrics-tls\") pod \"dns-default-jq8v5\" (UID: \"1e652714-955b-4d9d-a768-f781cf121cea\") " pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:14:57.757948 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:57.757923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rpdf\""
Jul 29 01:14:57.766062 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:57.766027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:14:57.888132 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:57.888099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jq8v5"]
Jul 29 01:14:57.890988 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:14:57.890965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e652714_955b_4d9d_a768_f781cf121cea.slice/crio-b45da07586061cc1ad2bb6109a4bc09fb3e312ad8ab47368795c37ab3e7d7c03 WatchSource:0}: Error finding container b45da07586061cc1ad2bb6109a4bc09fb3e312ad8ab47368795c37ab3e7d7c03: Status 404 returned error can't find the container with id b45da07586061cc1ad2bb6109a4bc09fb3e312ad8ab47368795c37ab3e7d7c03
Jul 29 01:14:58.467846 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:58.467799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq8v5" event={"ID":"1e652714-955b-4d9d-a768-f781cf121cea","Type":"ContainerStarted","Data":"b45da07586061cc1ad2bb6109a4bc09fb3e312ad8ab47368795c37ab3e7d7c03"}
Jul 29 01:14:59.471698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:59.471665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq8v5" event={"ID":"1e652714-955b-4d9d-a768-f781cf121cea","Type":"ContainerStarted","Data":"fbf083d6d4278bf3892b8368020139ff22f2e1e88a05959847833724844ddaa6"}
Jul 29 01:14:59.472001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:59.471723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq8v5" event={"ID":"1e652714-955b-4d9d-a768-f781cf121cea","Type":"ContainerStarted","Data":"4a2167fb0c63fc4df5dc6e25f2b6837badebc5a56559893c2a9c09a6a76bd869"}
Jul 29 01:14:59.472001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:59.471769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:14:59.489190 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:14:59.489150    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jq8v5" podStartSLOduration=251.094415127 podStartE2EDuration="4m12.489137629s" totalImagesPullingTime="1.394722502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:14:59.487526498 +0000 UTC m=+286.640971874" watchObservedRunningTime="2026-07-29 01:14:59.489137629 +0000 UTC m=+286.642583005"
Jul 29 01:15:08.134250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:08.134219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:15:08.136488 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:08.136469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8fc14f1-4a24-4d26-94d4-4c814c404f22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z2kk\" (UID: \"c8fc14f1-4a24-4d26-94d4-4c814c404f22\") " pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:15:08.385267 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:08.385183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2z2kk"
Jul 29 01:15:08.462181 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:15:08.462081    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ctvcr" podUID="379367f7-b885-4486-b103-e9d470325865"
Jul 29 01:15:08.508830 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:08.508790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2z2kk"]
Jul 29 01:15:08.511526 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:08.511502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fc14f1_4a24_4d26_94d4_4c814c404f22.slice/crio-3b91448e34897468a7929df2f86e21f787d1e4c923a581a1436c6fabe71992ff WatchSource:0}: Error finding container 3b91448e34897468a7929df2f86e21f787d1e4c923a581a1436c6fabe71992ff: Status 404 returned error can't find the container with id 3b91448e34897468a7929df2f86e21f787d1e4c923a581a1436c6fabe71992ff
Jul 29 01:15:09.477660 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:09.477627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jq8v5"
Jul 29 01:15:09.500192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:09.500159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z2kk" event={"ID":"c8fc14f1-4a24-4d26-94d4-4c814c404f22","Type":"ContainerStarted","Data":"3b91448e34897468a7929df2f86e21f787d1e4c923a581a1436c6fabe71992ff"}
Jul 29 01:15:13.472777 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:13.472653    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 01:15:14.517889 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:14.517847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z2kk" event={"ID":"c8fc14f1-4a24-4d26-94d4-4c814c404f22","Type":"ContainerStarted","Data":"b13a2cd5d6dcbd855c62ac5070e97ecc028afdde86142e60f17206a2d22f6221"}
Jul 29 01:15:16.387973 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.387700    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"]
Jul 29 01:15:16.396410 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.396382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.400372 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.400348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 01:15:16.401562 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.401542    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 01:15:16.402252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.402068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-97f79\""
Jul 29 01:15:16.402895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.402658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 01:15:16.402978 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.402896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 01:15:16.403959 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.403941    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 01:15:16.408485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.408467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"]
Jul 29 01:15:16.445890 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.445867    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bf4bl"]
Jul 29 01:15:16.450867 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.450847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.453935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.453896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pnbbx\""
Jul 29 01:15:16.454078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.454023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 01:15:16.454282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.454260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 01:15:16.454395 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.454310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 01:15:16.497439 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.497411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8jq\" (UniqueName: \"kubernetes.io/projected/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-kube-api-access-5k8jq\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.497618 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.497447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.497618 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.497510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.497618 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.497537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.598451 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8jq\" (UniqueName: \"kubernetes.io/projected/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-kube-api-access-5k8jq\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.598628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.598628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-wtmp\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.598628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-root\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-metrics-client-ca\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-textfile\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/9f979293-3c07-4821-9763-0166f1e82ed0-kube-api-access-5cvtt\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.598905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.599208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.598929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-sys\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.599298 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.599273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.601574 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.601545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.601748 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.601647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.607925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.607905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8jq\" (UniqueName: \"kubernetes.io/projected/ce35a5fd-22ff-4427-b846-01bbcaa8cd9e-kube-api-access-5k8jq\") pod \"openshift-state-metrics-b8f49f5c6-t6r6z\" (UID: \"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.700254 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700254 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-root\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700254 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-metrics-client-ca\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:15:16.700267    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-root\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:15:16.700350    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls podName:9f979293-3c07-4821-9763-0166f1e82ed0 nodeName:}" failed. No retries permitted until 2026-07-29 01:15:17.200334276 +0000 UTC m=+304.353779634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls") pod "node-exporter-bf4bl" (UID: "9f979293-3c07-4821-9763-0166f1e82ed0") : secret "node-exporter-tls" not found
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-textfile\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/9f979293-3c07-4821-9763-0166f1e82ed0-kube-api-access-5cvtt\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-sys\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-wtmp\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-sys\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-textfile\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-wtmp\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.700921 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.700908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-metrics-client-ca\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.701202 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.701156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.703217 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.703180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:16.707143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.707122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"
Jul 29 01:15:16.715611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:16.715592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/9f979293-3c07-4821-9763-0166f1e82ed0-kube-api-access-5cvtt\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:17.006764 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.006739    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z"]
Jul 29 01:15:17.007577 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:17.007554    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce35a5fd_22ff_4427_b846_01bbcaa8cd9e.slice/crio-2252386e36df6159e0f2078773343958ab9f1518f751c9550626b52336515265 WatchSource:0}: Error finding container 2252386e36df6159e0f2078773343958ab9f1518f751c9550626b52336515265: Status 404 returned error can't find the container with id 2252386e36df6159e0f2078773343958ab9f1518f751c9550626b52336515265
Jul 29 01:15:17.007869 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.007851    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:15:17.205329 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.205299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:17.207530 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.207510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f979293-3c07-4821-9763-0166f1e82ed0-node-exporter-tls\") pod \"node-exporter-bf4bl\" (UID: \"9f979293-3c07-4821-9763-0166f1e82ed0\") " pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:17.361139 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.361107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bf4bl"
Jul 29 01:15:17.372319 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:17.372294    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f979293_3c07_4821_9763_0166f1e82ed0.slice/crio-309b1ca4ba2b4d1509ede398a2b86da450a51ed54558773868736d8056f6e874 WatchSource:0}: Error finding container 309b1ca4ba2b4d1509ede398a2b86da450a51ed54558773868736d8056f6e874: Status 404 returned error can't find the container with id 309b1ca4ba2b4d1509ede398a2b86da450a51ed54558773868736d8056f6e874
Jul 29 01:15:17.420800 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.420767    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:15:17.425856 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.425834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.428445 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 01:15:17.428445 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 01:15:17.428445 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 01:15:17.428445 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 01:15:17.428445 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 01:15:17.428746 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428677    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 01:15:17.428801 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 01:15:17.428853 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.428802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 01:15:17.429026 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.429009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 01:15:17.429129 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.429087    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-brfg7\""
Jul 29 01:15:17.440792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.440774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:15:17.533427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.533391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bf4bl" event={"ID":"9f979293-3c07-4821-9763-0166f1e82ed0","Type":"ContainerStarted","Data":"309b1ca4ba2b4d1509ede398a2b86da450a51ed54558773868736d8056f6e874"}
Jul 29 01:15:17.535296 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.535269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z2kk" event={"ID":"c8fc14f1-4a24-4d26-94d4-4c814c404f22","Type":"ContainerStarted","Data":"5b89826124ea5960bcde7b2ddf9ddb0fe44611706433e64b4890052de3e13d47"}
Jul 29 01:15:17.537175 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.537154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z" event={"ID":"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e","Type":"ContainerStarted","Data":"07d9d97bbd6dc674dfa609263e9ad4d08306bdf570e882331b7d5e0d1b82830b"}
Jul 29 01:15:17.537291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.537195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z" event={"ID":"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e","Type":"ContainerStarted","Data":"875e90b8d6b740c57131f4a830559bcfeeaeda0ef40d4e9df2599040868a0291"}
Jul 29 01:15:17.537291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.537217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z" event={"ID":"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e","Type":"ContainerStarted","Data":"2252386e36df6159e0f2078773343958ab9f1518f751c9550626b52336515265"}
Jul 29 01:15:17.608816 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.608778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.608985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.608862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.608985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.608921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.608985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.608956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwvp\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.609371 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.609344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710692 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.710847 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwvp\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.710990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.711192 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.711014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.712011 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.711603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.713269 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.712418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.713269 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.712956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.714637 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.714609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.715420 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.715384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.716121 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.716008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.716121 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.716034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.716436 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.716417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.716519 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.716490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.716576 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.716528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.717088 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.717066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.718588 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.718221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.720985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.720959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwvp\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp\") pod \"alertmanager-main-0\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.738020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.737820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:15:17.897970 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:17.897916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:15:18.134709 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:18.134675    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a00aad_0ac9_4783_97b9_f349e103fdf9.slice/crio-a7f501feeb5985643213386f50cf36b9d65b24412ecf3c586703ce7d39ad1709 WatchSource:0}: Error finding container a7f501feeb5985643213386f50cf36b9d65b24412ecf3c586703ce7d39ad1709: Status 404 returned error can't find the container with id a7f501feeb5985643213386f50cf36b9d65b24412ecf3c586703ce7d39ad1709
Jul 29 01:15:18.543005 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:18.542920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"a7f501feeb5985643213386f50cf36b9d65b24412ecf3c586703ce7d39ad1709"}
Jul 29 01:15:18.545309 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:18.545271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z" event={"ID":"ce35a5fd-22ff-4427-b846-01bbcaa8cd9e","Type":"ContainerStarted","Data":"f7b6e5870c637386d15907330181ce9c3f97d14ed4a437d28675c89bd7385a38"}
Jul 29 01:15:18.566580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:18.566537    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-t6r6z" podStartSLOduration=1.495695647 podStartE2EDuration="2.566519672s" totalImagesPullingTime="1.070824025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:18.564712589 +0000 UTC m=+305.718157968" watchObservedRunningTime="2026-07-29 01:15:18.566519672 +0000 UTC m=+305.719965049"
Jul 29 01:15:19.411611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.411578    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b94868948-jk4kd"]
Jul 29 01:15:19.422346 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.422320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.426411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 01:15:19.426411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 01:15:19.426411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 01:15:19.426411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c755jf028nuoi\""
Jul 29 01:15:19.426411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 01:15:19.426748 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.426531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rns4z\""
Jul 29 01:15:19.427531 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.427462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 01:15:19.429023 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.428970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b94868948-jk4kd"]
Jul 29 01:15:19.526235 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4cl\" (UniqueName: \"kubernetes.io/projected/f4e6eff0-2779-4d31-8967-0139e359c218-kube-api-access-mp4cl\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-grpc-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4e6eff0-2779-4d31-8967-0139e359c218-metrics-client-ca\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.526614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.526524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.552459 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.552421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bf4bl" event={"ID":"9f979293-3c07-4821-9763-0166f1e82ed0","Type":"ContainerStarted","Data":"9ee57ad8ed0740307bd3b7b2e3902cc7383afce1913c33b163fbb700d4908b51"}
Jul 29 01:15:19.627287 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4e6eff0-2779-4d31-8967-0139e359c218-metrics-client-ca\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4cl\" (UniqueName: \"kubernetes.io/projected/f4e6eff0-2779-4d31-8967-0139e359c218-kube-api-access-mp4cl\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627695 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.627695 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-grpc-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.628291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.627711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.628713 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.628463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4e6eff0-2779-4d31-8967-0139e359c218-metrics-client-ca\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.630446 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.630402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.630547 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.630485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.630970 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.630948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.631094 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.631016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-grpc-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.631094 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.631070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.632418 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.632392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4e6eff0-2779-4d31-8967-0139e359c218-secret-thanos-querier-tls\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.636691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.636674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4cl\" (UniqueName: \"kubernetes.io/projected/f4e6eff0-2779-4d31-8967-0139e359c218-kube-api-access-mp4cl\") pod \"thanos-querier-5b94868948-jk4kd\" (UID: \"f4e6eff0-2779-4d31-8967-0139e359c218\") " pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:19.733929 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:19.733868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:20.557421 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:20.557389    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f979293-3c07-4821-9763-0166f1e82ed0" containerID="9ee57ad8ed0740307bd3b7b2e3902cc7383afce1913c33b163fbb700d4908b51" exitCode=0
Jul 29 01:15:20.557791 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:20.557462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bf4bl" event={"ID":"9f979293-3c07-4821-9763-0166f1e82ed0","Type":"ContainerDied","Data":"9ee57ad8ed0740307bd3b7b2e3902cc7383afce1913c33b163fbb700d4908b51"}
Jul 29 01:15:20.951391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:20.951368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b94868948-jk4kd"]
Jul 29 01:15:21.000214 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:21.000188    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e6eff0_2779_4d31_8967_0139e359c218.slice/crio-37a3023b798df25c991c1e4ab2df80eea5d37a7a4357bf5343fcb6f6d0cfc30b WatchSource:0}: Error finding container 37a3023b798df25c991c1e4ab2df80eea5d37a7a4357bf5343fcb6f6d0cfc30b: Status 404 returned error can't find the container with id 37a3023b798df25c991c1e4ab2df80eea5d37a7a4357bf5343fcb6f6d0cfc30b
Jul 29 01:15:21.563587 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.563552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z2kk" event={"ID":"c8fc14f1-4a24-4d26-94d4-4c814c404f22","Type":"ContainerStarted","Data":"88a7b0af78a7cb81060b1f87f30689e8186bcd2519db5f2a1f1915ba0750b396"}
Jul 29 01:15:21.564945 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.564913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"37a3023b798df25c991c1e4ab2df80eea5d37a7a4357bf5343fcb6f6d0cfc30b"}
Jul 29 01:15:21.567340 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.567313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bf4bl" event={"ID":"9f979293-3c07-4821-9763-0166f1e82ed0","Type":"ContainerStarted","Data":"68a1e1857022e9275e52c770f1fd9795ab082eda3c686b5bae11c816093349df"}
Jul 29 01:15:21.567449 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.567423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bf4bl" event={"ID":"9f979293-3c07-4821-9763-0166f1e82ed0","Type":"ContainerStarted","Data":"80ae13eb45e89e57fac19cf9c0afb2c597c2d5c0d67329534faeacc59a5def4c"}
Jul 29 01:15:21.568878 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.568855    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="8eb3bdc2f9fc0999d55307991c5f723422b39d47df1b864656bda56c1f89fbfe" exitCode=0
Jul 29 01:15:21.568960 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.568882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"8eb3bdc2f9fc0999d55307991c5f723422b39d47df1b864656bda56c1f89fbfe"}
Jul 29 01:15:21.582614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.582543    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2z2kk" podStartSLOduration=33.401807876 podStartE2EDuration="45.582529536s" totalImagesPullingTime="12.18072166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:14:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:21.581536149 +0000 UTC m=+308.734981527" watchObservedRunningTime="2026-07-29 01:15:21.582529536 +0000 UTC m=+308.735974911"
Jul 29 01:15:21.600113 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:21.600075    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bf4bl" podStartSLOduration=4.115233844 podStartE2EDuration="5.600062404s" totalImagesPullingTime="1.415917614s" totalInitContainerRuntime="68.910946ms" isStatefulPod=true podCreationTimestamp="2026-07-29 01:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:21.599334865 +0000 UTC m=+308.752780243" watchObservedRunningTime="2026-07-29 01:15:21.600062404 +0000 UTC m=+308.753507771"
Jul 29 01:15:22.461058 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:22.461011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:15:23.577733 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.577702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"317954c5e0dc90ea58a52f62a8cd375b521064e7b92ec3e4eec5d25707054320"}
Jul 29 01:15:23.578165 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.577769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"cffc05dfce0d672b201a69f8fc6363f47702046e08bec87177955c4c79e1b6da"}
Jul 29 01:15:23.578165 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.577801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"1dbc3943d2a36c21144ed26e288104fb9ff9193d15e8d5230139adf185867fb1"}
Jul 29 01:15:23.580581 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.580554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"cad84a6526dcbab05e7306d86c7e2c7e2568cc02716b8bc9ae2fcd0129182f47"}
Jul 29 01:15:23.580665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.580610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"f04eaf5e495c1f06dde2d7c8d160d0d5007e3cf5eb0e6380b7bacbb36f3e6a9b"}
Jul 29 01:15:23.580665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.580635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"c804c7b362c7a5c5e13538566487b238a6af2e27cec2cd944fe5cd24dc95d71e"}
Jul 29 01:15:23.580665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.580659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"1e34aed048730d596585642dcde5e91cf4bf666256f27375b9fe266afe370f42"}
Jul 29 01:15:23.580776 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:23.580679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"0801d44833ab726e5a48b9715446aea59a023afd03fb63b57915db2ce305ad31"}
Jul 29 01:15:24.586953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.586918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerStarted","Data":"11a425f8530d67a0bc2f5784a269e841ff93822af2e6ca44f639cf3360bb9ff3"}
Jul 29 01:15:24.589831 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.589805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"5366312fc9037cece98b36f30fbbc6edf2568421ee80ab7627237f05531f33f6"}
Jul 29 01:15:24.589930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.589857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"98e14be1b3f952557ba5d9ce09b9c29f804f8d791e4e2f5a6f9fa18be3da295a"}
Jul 29 01:15:24.589930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.589883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" event={"ID":"f4e6eff0-2779-4d31-8967-0139e359c218","Type":"ContainerStarted","Data":"f464d2e6651fce6e7598685e27738b15da24fd956f7d0577981187d6dab40c79"}
Jul 29 01:15:24.589930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.589922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:24.614210 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.614171    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5820779270000003 podStartE2EDuration="7.614156819s" totalImagesPullingTime="5.004298458s" totalInitContainerRuntime="27.780434ms" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:24.611856137 +0000 UTC m=+311.765301513" watchObservedRunningTime="2026-07-29 01:15:24.614156819 +0000 UTC m=+311.767602198"
Jul 29 01:15:24.635851 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:24.635801    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd" podStartSLOduration=2.631079464 podStartE2EDuration="5.635783193s" totalImagesPullingTime="3.004703729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:24.634492195 +0000 UTC m=+311.787937572" watchObservedRunningTime="2026-07-29 01:15:24.635783193 +0000 UTC m=+311.789228571"
Jul 29 01:15:29.136536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.136506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-tpbnn"]
Jul 29 01:15:29.139994 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.139978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:29.144712 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.144693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xv7jq\""
Jul 29 01:15:29.144967 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.144945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 01:15:29.144967 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.144958    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 01:15:29.152383 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.152364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-tpbnn"]
Jul 29 01:15:29.203179 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.203158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j979l\" (UniqueName: \"kubernetes.io/projected/ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83-kube-api-access-j979l\") pod \"downloads-7c97dc4555-tpbnn\" (UID: \"ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83\") " pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:29.303934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.303909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j979l\" (UniqueName: \"kubernetes.io/projected/ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83-kube-api-access-j979l\") pod \"downloads-7c97dc4555-tpbnn\" (UID: \"ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83\") " pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:29.312097 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.312069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j979l\" (UniqueName: \"kubernetes.io/projected/ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83-kube-api-access-j979l\") pod \"downloads-7c97dc4555-tpbnn\" (UID: \"ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83\") " pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:29.448230 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.448170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:29.569962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.569934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-tpbnn"]
Jul 29 01:15:29.573558 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:15:29.573530    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6acb4d_a8b5_48ce_88f3_f7b7a9a01a83.slice/crio-4ec6bd5b81cd883f6728e034baf7cc450ca54beba9056a19a99c94ea6d472eca WatchSource:0}: Error finding container 4ec6bd5b81cd883f6728e034baf7cc450ca54beba9056a19a99c94ea6d472eca: Status 404 returned error can't find the container with id 4ec6bd5b81cd883f6728e034baf7cc450ca54beba9056a19a99c94ea6d472eca
Jul 29 01:15:29.607928 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:29.607886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-tpbnn" event={"ID":"ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83","Type":"ContainerStarted","Data":"4ec6bd5b81cd883f6728e034baf7cc450ca54beba9056a19a99c94ea6d472eca"}
Jul 29 01:15:30.602196 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:30.602170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b94868948-jk4kd"
Jul 29 01:15:38.391907 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:38.391762    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" podUID="c4fabdd6-e856-437b-99ca-061b36ceb0fc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 01:15:47.687966 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:47.687920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-tpbnn" event={"ID":"ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83","Type":"ContainerStarted","Data":"a8ba06b6446d579dbb9c2dc0362268dfc12ea316023ac3c152f16693ed071ec5"}
Jul 29 01:15:47.688479 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:47.688164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:47.699647 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:47.699618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-tpbnn"
Jul 29 01:15:47.706911 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:47.706861    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-tpbnn" podStartSLOduration=1.121678262 podStartE2EDuration="18.706844115s" totalImagesPullingTime="17.585165853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:15:47.70532199 +0000 UTC m=+334.858767379" watchObservedRunningTime="2026-07-29 01:15:47.706844115 +0000 UTC m=+334.860289491"
Jul 29 01:15:48.391759 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:48.391706    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" podUID="c4fabdd6-e856-437b-99ca-061b36ceb0fc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 01:15:56.111739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:56.111709    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xnxd6_1fd6be0d-2df3-4faf-9974-ca1d237b2c33/serve-healthcheck-canary/0.log"
Jul 29 01:15:58.391640 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.391600    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" podUID="c4fabdd6-e856-437b-99ca-061b36ceb0fc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 01:15:58.391992 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.391676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7"
Jul 29 01:15:58.392621 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.392583    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"63d2b8bf031dabb9c412419d3df17fb133db5c80bacb52f05f27e9c3a8789776"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 29 01:15:58.392718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.392638    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" podUID="c4fabdd6-e856-437b-99ca-061b36ceb0fc" containerName="service-proxy" containerID="cri-o://63d2b8bf031dabb9c412419d3df17fb133db5c80bacb52f05f27e9c3a8789776" gracePeriod=30
Jul 29 01:15:58.732407 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.732324    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4fabdd6-e856-437b-99ca-061b36ceb0fc" containerID="63d2b8bf031dabb9c412419d3df17fb133db5c80bacb52f05f27e9c3a8789776" exitCode=2
Jul 29 01:15:58.732407 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:58.732365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerDied","Data":"63d2b8bf031dabb9c412419d3df17fb133db5c80bacb52f05f27e9c3a8789776"}
Jul 29 01:15:59.737939 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:15:59.737902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d6d6fd84-hnkn7" event={"ID":"c4fabdd6-e856-437b-99ca-061b36ceb0fc","Type":"ContainerStarted","Data":"3fa19ae33fbe3fcc8a7abb3ddfe26df899435515eafc085f632d1685e9cf1779"}
Jul 29 01:16:26.284168 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.284079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:16:26.286494 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.286467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379367f7-b885-4486-b103-e9d470325865-metrics-certs\") pod \"network-metrics-daemon-ctvcr\" (UID: \"379367f7-b885-4486-b103-e9d470325865\") " pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:16:26.364246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.364220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-llbnf\""
Jul 29 01:16:26.371710 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.371686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctvcr"
Jul 29 01:16:26.529082 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.529059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctvcr"]
Jul 29 01:16:26.531111 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:16:26.531085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379367f7_b885_4486_b103_e9d470325865.slice/crio-be389d105b8a2a20171f3af757775774e4d9f5e3599047b930ce1034672ed220 WatchSource:0}: Error finding container be389d105b8a2a20171f3af757775774e4d9f5e3599047b930ce1034672ed220: Status 404 returned error can't find the container with id be389d105b8a2a20171f3af757775774e4d9f5e3599047b930ce1034672ed220
Jul 29 01:16:26.837438 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:26.837408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctvcr" event={"ID":"379367f7-b885-4486-b103-e9d470325865","Type":"ContainerStarted","Data":"be389d105b8a2a20171f3af757775774e4d9f5e3599047b930ce1034672ed220"}
Jul 29 01:16:28.847135 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:28.847094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctvcr" event={"ID":"379367f7-b885-4486-b103-e9d470325865","Type":"ContainerStarted","Data":"742b3a0f55f118cdfa929617d076c5c8606a87b1ff1cd69a6ab3488c3e51ba2a"}
Jul 29 01:16:28.847563 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:28.847193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctvcr" event={"ID":"379367f7-b885-4486-b103-e9d470325865","Type":"ContainerStarted","Data":"bb99fb048d13c603e1c1e8075673758bf69ca4ec4b422641ba0ae10e6b6876eb"}
Jul 29 01:16:28.865636 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:28.865596    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ctvcr" podStartSLOduration=374.511806187 podStartE2EDuration="6m15.865583422s" totalImagesPullingTime="1.353777235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:28.864458456 +0000 UTC m=+376.017903830" watchObservedRunningTime="2026-07-29 01:16:28.865583422 +0000 UTC m=+376.019028798"
Jul 29 01:16:36.663786 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.663754    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:36.664258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664067    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="alertmanager" containerID="cri-o://0801d44833ab726e5a48b9715446aea59a023afd03fb63b57915db2ce305ad31" gracePeriod=120
Jul 29 01:16:36.664258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664112    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-metric" containerID="cri-o://cad84a6526dcbab05e7306d86c7e2c7e2568cc02716b8bc9ae2fcd0129182f47" gracePeriod=120
Jul 29 01:16:36.664258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664147    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-web" containerID="cri-o://c804c7b362c7a5c5e13538566487b238a6af2e27cec2cd944fe5cd24dc95d71e" gracePeriod=120
Jul 29 01:16:36.664258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664182    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy" containerID="cri-o://f04eaf5e495c1f06dde2d7c8d160d0d5007e3cf5eb0e6380b7bacbb36f3e6a9b" gracePeriod=120
Jul 29 01:16:36.664258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664187    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="config-reloader" containerID="cri-o://1e34aed048730d596585642dcde5e91cf4bf666256f27375b9fe266afe370f42" gracePeriod=120
Jul 29 01:16:36.664498 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.664132    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="prom-label-proxy" containerID="cri-o://11a425f8530d67a0bc2f5784a269e841ff93822af2e6ca44f639cf3360bb9ff3" gracePeriod=120
Jul 29 01:16:36.880250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880220    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="11a425f8530d67a0bc2f5784a269e841ff93822af2e6ca44f639cf3360bb9ff3" exitCode=0
Jul 29 01:16:36.880250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880245    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="f04eaf5e495c1f06dde2d7c8d160d0d5007e3cf5eb0e6380b7bacbb36f3e6a9b" exitCode=0
Jul 29 01:16:36.880250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880251    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="1e34aed048730d596585642dcde5e91cf4bf666256f27375b9fe266afe370f42" exitCode=0
Jul 29 01:16:36.880250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880257    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="0801d44833ab726e5a48b9715446aea59a023afd03fb63b57915db2ce305ad31" exitCode=0
Jul 29 01:16:36.880525 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"11a425f8530d67a0bc2f5784a269e841ff93822af2e6ca44f639cf3360bb9ff3"}
Jul 29 01:16:36.880525 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"f04eaf5e495c1f06dde2d7c8d160d0d5007e3cf5eb0e6380b7bacbb36f3e6a9b"}
Jul 29 01:16:36.880525 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"1e34aed048730d596585642dcde5e91cf4bf666256f27375b9fe266afe370f42"}
Jul 29 01:16:36.880525 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:36.880416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"0801d44833ab726e5a48b9715446aea59a023afd03fb63b57915db2ce305ad31"}
Jul 29 01:16:37.888147 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.888113    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="cad84a6526dcbab05e7306d86c7e2c7e2568cc02716b8bc9ae2fcd0129182f47" exitCode=0
Jul 29 01:16:37.888147 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.888144    2577 generic.go:356] "Generic (PLEG): container finished" podID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerID="c804c7b362c7a5c5e13538566487b238a6af2e27cec2cd944fe5cd24dc95d71e" exitCode=0
Jul 29 01:16:37.888560 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.888190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"cad84a6526dcbab05e7306d86c7e2c7e2568cc02716b8bc9ae2fcd0129182f47"}
Jul 29 01:16:37.888560 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.888278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"c804c7b362c7a5c5e13538566487b238a6af2e27cec2cd944fe5cd24dc95d71e"}
Jul 29 01:16:37.911722 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.911702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:37.976553 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976553 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976556    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrwvp\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.976761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.976907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls\") pod \"10a00aad-0ac9-4783-97b9-f349e103fdf9\" (UID: \"10a00aad-0ac9-4783-97b9-f349e103fdf9\") "
Jul 29 01:16:37.977328 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.977092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:37.977803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.977770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:16:37.980139 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.979860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.980449 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.980419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:16:37.980866 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.980836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.980866 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.980853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.980999 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.980841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:37.981352 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.981329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.981434 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.981368    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp" (OuterVolumeSpecName: "kube-api-access-qrwvp") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "kube-api-access-qrwvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:16:37.981913 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.981895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out" (OuterVolumeSpecName: "config-out") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:16:37.982520 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.982504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume" (OuterVolumeSpecName: "config-volume") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.985004 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.984981    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:37.990401 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:37.990380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config" (OuterVolumeSpecName: "web-config") pod "10a00aad-0ac9-4783-97b9-f349e103fdf9" (UID: "10a00aad-0ac9-4783-97b9-f349e103fdf9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:16:38.077900 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077869    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-metrics-client-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.077900 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077898    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-main-tls\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077912    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-tls-assets\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077927    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077941    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwvp\" (UniqueName: \"kubernetes.io/projected/10a00aad-0ac9-4783-97b9-f349e103fdf9-kube-api-access-qrwvp\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077953    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077965    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-cluster-tls-config\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.077976    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.078015    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-web-config\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.078027    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.078063    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-config-out\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.078076    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10a00aad-0ac9-4783-97b9-f349e103fdf9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.078127 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.078089    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10a00aad-0ac9-4783-97b9-f349e103fdf9-alertmanager-main-db\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:16:38.894551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.894511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10a00aad-0ac9-4783-97b9-f349e103fdf9","Type":"ContainerDied","Data":"a7f501feeb5985643213386f50cf36b9d65b24412ecf3c586703ce7d39ad1709"}
Jul 29 01:16:38.894551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.894540    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:38.895085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.894591    2577 scope.go:117] "RemoveContainer" containerID="11a425f8530d67a0bc2f5784a269e841ff93822af2e6ca44f639cf3360bb9ff3"
Jul 29 01:16:38.902535 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.902512    2577 scope.go:117] "RemoveContainer" containerID="cad84a6526dcbab05e7306d86c7e2c7e2568cc02716b8bc9ae2fcd0129182f47"
Jul 29 01:16:38.909825 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.909806    2577 scope.go:117] "RemoveContainer" containerID="f04eaf5e495c1f06dde2d7c8d160d0d5007e3cf5eb0e6380b7bacbb36f3e6a9b"
Jul 29 01:16:38.916124 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.916103    2577 scope.go:117] "RemoveContainer" containerID="c804c7b362c7a5c5e13538566487b238a6af2e27cec2cd944fe5cd24dc95d71e"
Jul 29 01:16:38.921596 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.921575    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:38.923832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.923808    2577 scope.go:117] "RemoveContainer" containerID="1e34aed048730d596585642dcde5e91cf4bf666256f27375b9fe266afe370f42"
Jul 29 01:16:38.925775 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.925757    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:38.930892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.930873    2577 scope.go:117] "RemoveContainer" containerID="0801d44833ab726e5a48b9715446aea59a023afd03fb63b57915db2ce305ad31"
Jul 29 01:16:38.937389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.937370    2577 scope.go:117] "RemoveContainer" containerID="8eb3bdc2f9fc0999d55307991c5f723422b39d47df1b864656bda56c1f89fbfe"
Jul 29 01:16:38.953571 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953551    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:38.953805 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953793    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="config-reloader"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953805    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="config-reloader"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953814    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="alertmanager"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953821    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="alertmanager"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953828    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-metric"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953834    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-metric"
Jul 29 01:16:38.953845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953848    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953855    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="prom-label-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953861    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="prom-label-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953867    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-web"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953872    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-web"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953880    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="init-config-reloader"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953885    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="init-config-reloader"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953923    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953929    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-metric"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953937    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="alertmanager"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953942    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="kube-rbac-proxy-web"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953947    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="prom-label-proxy"
Jul 29 01:16:38.954029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.953955    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" containerName="config-reloader"
Jul 29 01:16:38.957563 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.957548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:38.959773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.959757    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 01:16:38.959864 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.959809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 01:16:38.959864 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.959820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 01:16:38.960354 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960337    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 01:16:38.960423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 01:16:38.960423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 01:16:38.960526 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-brfg7\""
Jul 29 01:16:38.960690 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 01:16:38.960771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.960756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 01:16:38.965516 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.965498    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 01:16:38.974909 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:38.974884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:39.086993 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.086962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xzd\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-kube-api-access-j7xzd\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087288 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087288 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087288 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.087532 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.087432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188779 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188779 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188779 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xzd\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-kube-api-access-j7xzd\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188927 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188927 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188927 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.188927 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.188900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.189483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.189134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.189602 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.189562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.191547 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.191521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.191750 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.191709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.191980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b376f79-786c-4c1e-992e-bef99cae8dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.192608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.192475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.193509 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.193492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b376f79-786c-4c1e-992e-bef99cae8dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.200076 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.200055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xzd\" (UniqueName: \"kubernetes.io/projected/0b376f79-786c-4c1e-992e-bef99cae8dc1-kube-api-access-j7xzd\") pod \"alertmanager-main-0\" (UID: \"0b376f79-786c-4c1e-992e-bef99cae8dc1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.267055 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.267014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 01:16:39.410753 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.410718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 01:16:39.414372 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:16:39.414337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b376f79_786c_4c1e_992e_bef99cae8dc1.slice/crio-fcf7566176e53edb099898bb9d5fe7ba3f940948daeb6c4598a9ccba224f4ef3 WatchSource:0}: Error finding container fcf7566176e53edb099898bb9d5fe7ba3f940948daeb6c4598a9ccba224f4ef3: Status 404 returned error can't find the container with id fcf7566176e53edb099898bb9d5fe7ba3f940948daeb6c4598a9ccba224f4ef3
Jul 29 01:16:39.465201 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.465171    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a00aad-0ac9-4783-97b9-f349e103fdf9" path="/var/lib/kubelet/pods/10a00aad-0ac9-4783-97b9-f349e103fdf9/volumes"
Jul 29 01:16:39.900126 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.900093    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b376f79-786c-4c1e-992e-bef99cae8dc1" containerID="120e904a66f628be821912724b047009aa6bb9cf7f9ff16b70bfdd0bd4202f37" exitCode=0
Jul 29 01:16:39.900490 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.900153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerDied","Data":"120e904a66f628be821912724b047009aa6bb9cf7f9ff16b70bfdd0bd4202f37"}
Jul 29 01:16:39.900490 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:39.900204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"fcf7566176e53edb099898bb9d5fe7ba3f940948daeb6c4598a9ccba224f4ef3"}
Jul 29 01:16:40.700505 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.700471    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78cff774c6-fqsf7"]
Jul 29 01:16:40.702795 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.702779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.705588 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 29 01:16:40.705588 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705608    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 29 01:16:40.705841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tzqlp\""
Jul 29 01:16:40.705841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 29 01:16:40.705938 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 29 01:16:40.706012 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.705998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 29 01:16:40.710332 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.710310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 29 01:16:40.717292 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.717271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cff774c6-fqsf7"]
Jul 29 01:16:40.801841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.801972 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.801972 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-serving-certs-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.801972 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-metrics-client-ca\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.801972 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-federate-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.801972 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.802171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.801990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzh88\" (UniqueName: \"kubernetes.io/projected/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-kube-api-access-lzh88\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.802171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.802010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzh88\" (UniqueName: \"kubernetes.io/projected/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-kube-api-access-lzh88\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-serving-certs-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-metrics-client-ca\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.902953 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.902801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-federate-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.903586 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.903518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-serving-certs-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.903586 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.903549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-metrics-client-ca\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.903852 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.903801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.905360 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.905336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.905730 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.905707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-federate-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.905963 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.905930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-secret-telemeter-client\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.906281 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.906266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-telemeter-client-tls\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.908825 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"af62ed5c7a4a4da98a9edf47b7bede376a4cee334685567ca099bc7fad6a1325"}
Jul 29 01:16:40.908924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"eb245fff0594fab6c51874c25b80012aa67050c2dcbc7f7b7165fc4ec80037e3"}
Jul 29 01:16:40.908924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"312f7eedb1659c1b5c35898323b3db25b1d8f43c64a8df42396ff27f70320b04"}
Jul 29 01:16:40.908924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"9534c6fbaf71e539543845f4fd16d390d12c021f982241b5ee27087c65fcf4cf"}
Jul 29 01:16:40.909115 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"e7de4dc2569dde27a094ae0d3265d153edd8555abcd7070d50895593c2f92e39"}
Jul 29 01:16:40.909115 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.908957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b376f79-786c-4c1e-992e-bef99cae8dc1","Type":"ContainerStarted","Data":"262f22da9e2899149cf125117f50a204f2691795c66e5823c962076a2a56084b"}
Jul 29 01:16:40.911261 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.911242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzh88\" (UniqueName: \"kubernetes.io/projected/4ac4b421-15f1-458c-b4ee-571fb9ff9fbb-kube-api-access-lzh88\") pod \"telemeter-client-78cff774c6-fqsf7\" (UID: \"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb\") " pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:40.934076 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:40.934022    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.91893737 podStartE2EDuration="2.934008585s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.071215ms" isStatefulPod=false podCreationTimestamp="2026-07-29 01:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:40.933027339 +0000 UTC m=+388.086472728" watchObservedRunningTime="2026-07-29 01:16:40.934008585 +0000 UTC m=+388.087454008"
Jul 29 01:16:41.011861 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:41.011805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7"
Jul 29 01:16:41.139880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:41.139852    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cff774c6-fqsf7"]
Jul 29 01:16:41.142127 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:16:41.142100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac4b421_15f1_458c_b4ee_571fb9ff9fbb.slice/crio-7d042eed289a53ff57adb9f7e730e5301a92143e73e25ec60097a72096a44c38 WatchSource:0}: Error finding container 7d042eed289a53ff57adb9f7e730e5301a92143e73e25ec60097a72096a44c38: Status 404 returned error can't find the container with id 7d042eed289a53ff57adb9f7e730e5301a92143e73e25ec60097a72096a44c38
Jul 29 01:16:41.915209 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:41.915175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7" event={"ID":"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb","Type":"ContainerStarted","Data":"7d042eed289a53ff57adb9f7e730e5301a92143e73e25ec60097a72096a44c38"}
Jul 29 01:16:43.925702 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:43.925664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7" event={"ID":"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb","Type":"ContainerStarted","Data":"17bd2725039d48ed3197bc500f9ee27bf26856ea74c8506473af7e63ae2fc654"}
Jul 29 01:16:43.926124 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:43.925720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7" event={"ID":"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb","Type":"ContainerStarted","Data":"077a4535f332e774823233b7e35a180ab14903a17233ab167f0762b07d96369b"}
Jul 29 01:16:43.926124 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:43.925743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7" event={"ID":"4ac4b421-15f1-458c-b4ee-571fb9ff9fbb","Type":"ContainerStarted","Data":"89ee3687ebb8c8804e2ce847f1338afa2ad42f329119ac5148c840366c4663b0"}
Jul 29 01:16:43.954790 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:16:43.954739    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78cff774c6-fqsf7" podStartSLOduration=2.197046537 podStartE2EDuration="3.954722209s" totalImagesPullingTime="1.757675672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:16:43.95319535 +0000 UTC m=+391.106640725" watchObservedRunningTime="2026-07-29 01:16:43.954722209 +0000 UTC m=+391.108167584"
Jul 29 01:19:33.905652 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.905562    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-78n6h"]
Jul 29 01:19:33.908912 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.908889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:33.911974 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.911953    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 01:19:33.912093 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.911983    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 01:19:33.913087 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.913071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9rzcs\""
Jul 29 01:19:33.920317 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:33.920295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-78n6h"]
Jul 29 01:19:34.046387 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.046364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhx8r\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-kube-api-access-rhx8r\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.046485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.046400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-bound-sa-token\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.147674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.147641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-bound-sa-token\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.147842 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.147706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhx8r\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-kube-api-access-rhx8r\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.160353 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.157877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-bound-sa-token\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.160353 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.158034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhx8r\" (UniqueName: \"kubernetes.io/projected/dcdcd44c-b4ac-4e01-abdd-996b845e972d-kube-api-access-rhx8r\") pod \"cert-manager-55b4d954d5-78n6h\" (UID: \"dcdcd44c-b4ac-4e01-abdd-996b845e972d\") " pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.217548 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.217520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-78n6h"
Jul 29 01:19:34.344642 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.344618    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-78n6h"]
Jul 29 01:19:34.346355 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:19:34.346326    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcdcd44c_b4ac_4e01_abdd_996b845e972d.slice/crio-e3042eecf64e31da72bed0ece80f4cdff62ef5583d8d93852498c17ff08243e4 WatchSource:0}: Error finding container e3042eecf64e31da72bed0ece80f4cdff62ef5583d8d93852498c17ff08243e4: Status 404 returned error can't find the container with id e3042eecf64e31da72bed0ece80f4cdff62ef5583d8d93852498c17ff08243e4
Jul 29 01:19:34.560118 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:34.560029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-78n6h" event={"ID":"dcdcd44c-b4ac-4e01-abdd-996b845e972d","Type":"ContainerStarted","Data":"e3042eecf64e31da72bed0ece80f4cdff62ef5583d8d93852498c17ff08243e4"}
Jul 29 01:19:38.578980 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:38.578948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-78n6h" event={"ID":"dcdcd44c-b4ac-4e01-abdd-996b845e972d","Type":"ContainerStarted","Data":"720a1d267591513fff3216bf0ee8b1be0d3f2b528e94d8cb7a558f1fb4423b82"}
Jul 29 01:19:38.600513 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:19:38.600467    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-78n6h" podStartSLOduration=2.178863052 podStartE2EDuration="5.600452329s" totalImagesPullingTime="3.421589277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:19:38.599183684 +0000 UTC m=+565.752629060" watchObservedRunningTime="2026-07-29 01:19:38.600452329 +0000 UTC m=+565.753897705"
Jul 29 01:20:08.898589 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.898557    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"]
Jul 29 01:20:08.901732 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.901717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:08.909200 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.909178    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 01:20:08.909543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.909525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 01:20:08.909632 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.909572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 01:20:08.910447 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.910433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2h4qb\""
Jul 29 01:20:08.914324 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.914310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 01:20:08.952613 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:08.952589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"]
Jul 29 01:20:09.010334 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.010311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.010426 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.010337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfmm\" (UniqueName: \"kubernetes.io/projected/ff586751-2ef6-467b-9381-4b177ab766c3-kube-api-access-6xfmm\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.010426 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.010361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-tmp\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.010499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.010462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.010533 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.010505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-manifests\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.111651 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.111631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.111725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.111664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-manifests\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.111725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.111686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.111725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.111701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfmm\" (UniqueName: \"kubernetes.io/projected/ff586751-2ef6-467b-9381-4b177ab766c3-kube-api-access-6xfmm\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.111834 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.111726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-tmp\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.112089 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.112068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-manifests\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.113869 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.113844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff586751-2ef6-467b-9381-4b177ab766c3-tmp\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.113997 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.113979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.114078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.114018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff586751-2ef6-467b-9381-4b177ab766c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.121810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.121783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfmm\" (UniqueName: \"kubernetes.io/projected/ff586751-2ef6-467b-9381-4b177ab766c3-kube-api-access-6xfmm\") pod \"opendatahub-operator-controller-manager-857d7cdd4c-l25bs\" (UID: \"ff586751-2ef6-467b-9381-4b177ab766c3\") " pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.211527 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.211469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:09.344717 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.344693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"]
Jul 29 01:20:09.346980 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:20:09.346955    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff586751_2ef6_467b_9381_4b177ab766c3.slice/crio-d2ab32b938d9cc0a918a8b79a857f8c2bb5f726bfdab2c41eda0e6e1a2c4f710 WatchSource:0}: Error finding container d2ab32b938d9cc0a918a8b79a857f8c2bb5f726bfdab2c41eda0e6e1a2c4f710: Status 404 returned error can't find the container with id d2ab32b938d9cc0a918a8b79a857f8c2bb5f726bfdab2c41eda0e6e1a2c4f710
Jul 29 01:20:09.690323 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:09.690289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs" event={"ID":"ff586751-2ef6-467b-9381-4b177ab766c3","Type":"ContainerStarted","Data":"d2ab32b938d9cc0a918a8b79a857f8c2bb5f726bfdab2c41eda0e6e1a2c4f710"}
Jul 29 01:20:12.704198 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:12.704162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs" event={"ID":"ff586751-2ef6-467b-9381-4b177ab766c3","Type":"ContainerStarted","Data":"12cac6c748cadab357efa454940bd40ff1bbb35c17f2ac5d169f4be2028849d4"}
Jul 29 01:20:14.712795 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:14.712764    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff586751-2ef6-467b-9381-4b177ab766c3" containerID="12cac6c748cadab357efa454940bd40ff1bbb35c17f2ac5d169f4be2028849d4" exitCode=0
Jul 29 01:20:14.713211 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:14.712802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs" event={"ID":"ff586751-2ef6-467b-9381-4b177ab766c3","Type":"ContainerDied","Data":"12cac6c748cadab357efa454940bd40ff1bbb35c17f2ac5d169f4be2028849d4"}
Jul 29 01:20:15.718746 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:15.718715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs" event={"ID":"ff586751-2ef6-467b-9381-4b177ab766c3","Type":"ContainerStarted","Data":"c80bde8826b85020fbfe6ebc49c5ee523ff4bff9d79476fe8df9fbf875a19fb9"}
Jul 29 01:20:15.719152 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:15.718912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:26.725606 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:26.725577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs"
Jul 29 01:20:26.750375 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:26.750322    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-857d7cdd4c-l25bs" podStartSLOduration=14.070051755 podStartE2EDuration="18.750307237s" totalImagesPullingTime="3.603247576s" totalInitContainerRuntime="1.077007906s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:20:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:20:15.744465949 +0000 UTC m=+602.897911327" watchObservedRunningTime="2026-07-29 01:20:26.750307237 +0000 UTC m=+613.903752613"
Jul 29 01:20:29.179389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.179358    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"]
Jul 29 01:20:29.187567 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.187548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.191652 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.191631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 01:20:29.191753 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.191637    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 01:20:29.191851 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.191838    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 01:20:29.191895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.191878    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n4jwx\""
Jul 29 01:20:29.192365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.192349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 01:20:29.197098 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.197077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"]
Jul 29 01:20:29.261316 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.261286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tls-certs\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.261475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.261346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjncd\" (UniqueName: \"kubernetes.io/projected/7b46c7ae-7f5a-47ed-b546-84147b0f102d-kube-api-access-fjncd\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.261475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.261420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tmp\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.362678 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.362649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjncd\" (UniqueName: \"kubernetes.io/projected/7b46c7ae-7f5a-47ed-b546-84147b0f102d-kube-api-access-fjncd\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.362814 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.362686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tmp\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.362814 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.362710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tls-certs\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.364803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.364783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tmp\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.365135 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.365116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b46c7ae-7f5a-47ed-b546-84147b0f102d-tls-certs\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.377771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.377747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjncd\" (UniqueName: \"kubernetes.io/projected/7b46c7ae-7f5a-47ed-b546-84147b0f102d-kube-api-access-fjncd\") pod \"kube-auth-proxy-86657cdbcd-glh4b\" (UID: \"7b46c7ae-7f5a-47ed-b546-84147b0f102d\") " pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.497026 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.496959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"
Jul 29 01:20:29.645146 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.645113    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b"]
Jul 29 01:20:29.647253 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:20:29.647228    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b46c7ae_7f5a_47ed_b546_84147b0f102d.slice/crio-c31f2bd3b1516b39acea0b479d149cb568e19f0b6903a9748b4f7c424fe8e0c7 WatchSource:0}: Error finding container c31f2bd3b1516b39acea0b479d149cb568e19f0b6903a9748b4f7c424fe8e0c7: Status 404 returned error can't find the container with id c31f2bd3b1516b39acea0b479d149cb568e19f0b6903a9748b4f7c424fe8e0c7
Jul 29 01:20:29.649029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.649015    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:20:29.777746 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:29.777667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b" event={"ID":"7b46c7ae-7f5a-47ed-b546-84147b0f102d","Type":"ContainerStarted","Data":"c31f2bd3b1516b39acea0b479d149cb568e19f0b6903a9748b4f7c424fe8e0c7"}
Jul 29 01:20:30.364029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.363991    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tr5wm"]
Jul 29 01:20:30.367604 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.367580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.370163 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.370141    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-27h59\""
Jul 29 01:20:30.370324 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.370202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 29 01:20:30.383247 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.383227    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tr5wm"]
Jul 29 01:20:30.470612 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.470576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62bg\" (UniqueName: \"kubernetes.io/projected/d900390a-d129-4426-8525-5bbc998014da-kube-api-access-b62bg\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.470770 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.470650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900390a-d129-4426-8525-5bbc998014da-tls-certs\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.571784 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.571753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900390a-d129-4426-8525-5bbc998014da-tls-certs\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.571975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.571821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62bg\" (UniqueName: \"kubernetes.io/projected/d900390a-d129-4426-8525-5bbc998014da-kube-api-access-b62bg\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.574821 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.574795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900390a-d129-4426-8525-5bbc998014da-tls-certs\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.581123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.581102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62bg\" (UniqueName: \"kubernetes.io/projected/d900390a-d129-4426-8525-5bbc998014da-kube-api-access-b62bg\") pod \"model-serving-api-54d89bd5fc-tr5wm\" (UID: \"d900390a-d129-4426-8525-5bbc998014da\") " pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.684572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.684493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:30.828436 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:30.828402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tr5wm"]
Jul 29 01:20:30.831933 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:20:30.831902    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900390a_d129_4426_8525_5bbc998014da.slice/crio-f697dffb412a76bcf045a28b809f181e9fb1b15899ceafffdb973392b182f640 WatchSource:0}: Error finding container f697dffb412a76bcf045a28b809f181e9fb1b15899ceafffdb973392b182f640: Status 404 returned error can't find the container with id f697dffb412a76bcf045a28b809f181e9fb1b15899ceafffdb973392b182f640
Jul 29 01:20:31.789667 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:31.789632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm" event={"ID":"d900390a-d129-4426-8525-5bbc998014da","Type":"ContainerStarted","Data":"f697dffb412a76bcf045a28b809f181e9fb1b15899ceafffdb973392b182f640"}
Jul 29 01:20:32.795186 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:32.795086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm" event={"ID":"d900390a-d129-4426-8525-5bbc998014da","Type":"ContainerStarted","Data":"ea5006a525c8ca9addacf232df8ec41aec5bba387023df115a5639b506f929e7"}
Jul 29 01:20:32.795186 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:32.795152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:32.796509 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:32.796487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b" event={"ID":"7b46c7ae-7f5a-47ed-b546-84147b0f102d","Type":"ContainerStarted","Data":"670159f5c9b99f12561c23a6322d88451a24963cfc79c83cf2771f5bbf2bcae8"}
Jul 29 01:20:32.812810 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:32.812769    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm" podStartSLOduration=1.173001205 podStartE2EDuration="2.81275677s" totalImagesPullingTime="1.639755565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:20:32.812096969 +0000 UTC m=+619.965542345" watchObservedRunningTime="2026-07-29 01:20:32.81275677 +0000 UTC m=+619.966202145"
Jul 29 01:20:32.828093 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:32.828023    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86657cdbcd-glh4b" podStartSLOduration=1.002816641 podStartE2EDuration="3.82801014s" totalImagesPullingTime="2.825193499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:20:32.827497525 +0000 UTC m=+619.980942897" watchObservedRunningTime="2026-07-29 01:20:32.82801014 +0000 UTC m=+619.981455515"
Jul 29 01:20:36.680937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.680858    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:36.684341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.684321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.686816 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.686797    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 01:20:36.686932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.686913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zlvjd\""
Jul 29 01:20:36.695976 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.695955    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:36.725884 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.725858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.725903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.725925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.725972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.725995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.726018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.726059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867tx\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.726084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.726173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.726102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.826905 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.826868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867tx\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827106 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.826914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827106 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.826943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827106 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.826969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827106 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827106 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827393 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827393 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827393 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.827894 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.827872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.829341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.829313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.829650 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.829634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.837007 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.836988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.838311 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.838287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867tx\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcccxb5\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:36.994431 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:36.994339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:37.138564 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.138541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:37.140286 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:20:37.140256    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032f3af5_d437_4e8d_a541_b0d3bf7d22f3.slice/crio-d036b33657fc5d7b85f74ae87490757d54c14cddbb8d1107c13ea837a8caac4d WatchSource:0}: Error finding container d036b33657fc5d7b85f74ae87490757d54c14cddbb8d1107c13ea837a8caac4d: Status 404 returned error can't find the container with id d036b33657fc5d7b85f74ae87490757d54c14cddbb8d1107c13ea837a8caac4d
Jul 29 01:20:37.608391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.608349    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"]
Jul 29 01:20:37.612699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.612676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.616601 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.616576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 01:20:37.616736 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.616580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 01:20:37.616896 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.616878    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 01:20:37.616966 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.616951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 01:20:37.617074 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.617055    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jtn8v\""
Jul 29 01:20:37.617146 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.617127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 01:20:37.627409 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.627317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"]
Jul 29 01:20:37.746140 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.746107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrvf\" (UniqueName: \"kubernetes.io/projected/58729e2a-366b-4866-a52f-82dfe3801548-kube-api-access-hvrvf\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.746514 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.746169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58729e2a-366b-4866-a52f-82dfe3801548-manager-config\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.746514 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.746230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-metrics-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.746514 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.746287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.818765 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.818731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" event={"ID":"032f3af5-d437-4e8d-a541-b0d3bf7d22f3","Type":"ContainerStarted","Data":"d036b33657fc5d7b85f74ae87490757d54c14cddbb8d1107c13ea837a8caac4d"}
Jul 29 01:20:37.847346 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.847315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-metrics-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.847455 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.847353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.847455 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.847386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrvf\" (UniqueName: \"kubernetes.io/projected/58729e2a-366b-4866-a52f-82dfe3801548-kube-api-access-hvrvf\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.847455 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.847442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58729e2a-366b-4866-a52f-82dfe3801548-manager-config\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.848126 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.848108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58729e2a-366b-4866-a52f-82dfe3801548-manager-config\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.849914 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.849890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-metrics-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.850011 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.849890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58729e2a-366b-4866-a52f-82dfe3801548-cert\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.863115 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.863034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrvf\" (UniqueName: \"kubernetes.io/projected/58729e2a-366b-4866-a52f-82dfe3801548-kube-api-access-hvrvf\") pod \"lws-controller-manager-7c7f9c9bb4-ngq9z\" (UID: \"58729e2a-366b-4866-a52f-82dfe3801548\") " pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:37.923015 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:37.922994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:38.064337 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:38.064114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"]
Jul 29 01:20:38.066629 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:20:38.066599    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58729e2a_366b_4866_a52f_82dfe3801548.slice/crio-96318256f508d7e08807c5efc93d72be19a26019bfa378c5a7dc5724e5ba712b WatchSource:0}: Error finding container 96318256f508d7e08807c5efc93d72be19a26019bfa378c5a7dc5724e5ba712b: Status 404 returned error can't find the container with id 96318256f508d7e08807c5efc93d72be19a26019bfa378c5a7dc5724e5ba712b
Jul 29 01:20:38.824640 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:38.824600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z" event={"ID":"58729e2a-366b-4866-a52f-82dfe3801548","Type":"ContainerStarted","Data":"96318256f508d7e08807c5efc93d72be19a26019bfa378c5a7dc5724e5ba712b"}
Jul 29 01:20:40.074634 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.074593    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:20:40.074949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.074671    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:20:40.074949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.074708    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:20:40.834901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.834867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" event={"ID":"032f3af5-d437-4e8d-a541-b0d3bf7d22f3","Type":"ContainerStarted","Data":"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"}
Jul 29 01:20:40.901056 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.900991    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" podStartSLOduration=1.9690598590000001 podStartE2EDuration="4.900972156s" totalImagesPullingTime="2.931912297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:20:40.896976629 +0000 UTC m=+628.050422005" watchObservedRunningTime="2026-07-29 01:20:40.900972156 +0000 UTC m=+628.054417536"
Jul 29 01:20:40.994762 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:40.994727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:41.996133 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:41.996093    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Jul 29 01:20:41.996485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:41.996145    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Jul 29 01:20:42.848805 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:42.848770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z" event={"ID":"58729e2a-366b-4866-a52f-82dfe3801548","Type":"ContainerStarted","Data":"67809789415d95ff2b6cf49f104f0509219016d55db2a07fe0640c1d042b8de5"}
Jul 29 01:20:42.848967 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:42.848853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:20:42.871290 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:42.871243    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z" podStartSLOduration=2.07973722 podStartE2EDuration="5.871229215s" totalImagesPullingTime="3.791491995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:20:42.870747187 +0000 UTC m=+630.024192564" watchObservedRunningTime="2026-07-29 01:20:42.871229215 +0000 UTC m=+630.024674585"
Jul 29 01:20:42.994747 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:42.994715    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Jul 29 01:20:42.994912 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:42.994764    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Jul 29 01:20:43.804240 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:43.804212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tr5wm"
Jul 29 01:20:43.995610 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:43.995571    2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Jul 29 01:20:43.995802 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:43.995635    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Jul 29 01:20:44.277499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:44.277465    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:44.277674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:44.277645    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy" containerID="cri-o://55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b" gracePeriod=30
Jul 29 01:20:49.517565 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.517541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:49.649422 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649388    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649422 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867tx\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token\") pod \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\" (UID: \"032f3af5-d437-4e8d-a541-b0d3bf7d22f3\") "
Jul 29 01:20:49.649956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:20:49.649956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:20:49.649956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:20:49.649956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:20:49.650236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649961    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.650236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649981    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.650236 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.649978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:20:49.651975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.651946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 01:20:49.652101 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.651973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx" (OuterVolumeSpecName: "kube-api-access-867tx") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "kube-api-access-867tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:20:49.652162 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.652142    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:20:49.652210 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.652156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "032f3af5-d437-4e8d-a541-b0d3bf7d22f3" (UID: "032f3af5-d437-4e8d-a541-b0d3bf7d22f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:20:49.751213 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751182    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751213 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751207    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751213 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751216    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867tx\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-kube-api-access-867tx\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751233    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751241    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751248    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.751427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.751257    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032f3af5-d437-4e8d-a541-b0d3bf7d22f3-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:20:49.879771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.879737    2577 generic.go:356] "Generic (PLEG): container finished" podID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerID="55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b" exitCode=0
Jul 29 01:20:49.879934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.879797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"
Jul 29 01:20:49.879934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.879805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" event={"ID":"032f3af5-d437-4e8d-a541-b0d3bf7d22f3","Type":"ContainerDied","Data":"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"}
Jul 29 01:20:49.879934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.879873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5" event={"ID":"032f3af5-d437-4e8d-a541-b0d3bf7d22f3","Type":"ContainerDied","Data":"d036b33657fc5d7b85f74ae87490757d54c14cddbb8d1107c13ea837a8caac4d"}
Jul 29 01:20:49.879934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.879894    2577 scope.go:117] "RemoveContainer" containerID="55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"
Jul 29 01:20:49.888561 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.888541    2577 scope.go:117] "RemoveContainer" containerID="55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"
Jul 29 01:20:49.888812 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:20:49.888783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b\": container with ID starting with 55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b not found: ID does not exist" containerID="55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"
Jul 29 01:20:49.888862 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.888812    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b"} err="failed to get container status \"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b\": rpc error: code = NotFound desc = could not find container \"55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b\": container with ID starting with 55823fb2a954aed2e413e4a164c01de7a17a34e33e46b3c96f54e0998f97471b not found: ID does not exist"
Jul 29 01:20:49.900624 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.900568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:49.906189 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:49.906166    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcccxb5"]
Jul 29 01:20:51.466151 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:51.466123    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" path="/var/lib/kubelet/pods/032f3af5-d437-4e8d-a541-b0d3bf7d22f3/volumes"
Jul 29 01:20:53.855331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:20:53.855300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c7f9c9bb4-ngq9z"
Jul 29 01:23:25.569790 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.569756    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:23:25.570321 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.570090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy"
Jul 29 01:23:25.570321 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.570105    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy"
Jul 29 01:23:25.570321 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.570173    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="032f3af5-d437-4e8d-a541-b0d3bf7d22f3" containerName="istio-proxy"
Jul 29 01:23:25.572758 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.572741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.575424 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.575401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 01:23:25.575561 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.575401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 01:23:25.575561 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.575401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 01:23:25.576438 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.576423    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ls6zt\""
Jul 29 01:23:25.580241 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.580218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:23:25.613544 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.613511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:23:25.754243 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.754206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwv9\" (UniqueName: \"kubernetes.io/projected/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-kube-api-access-pvwv9\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.754416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.754265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-config-file\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.855751 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.855660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwv9\" (UniqueName: \"kubernetes.io/projected/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-kube-api-access-pvwv9\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.855751 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.855718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-config-file\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.856304 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.856288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-config-file\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.864669 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.864645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwv9\" (UniqueName: \"kubernetes.io/projected/8e9a7a8b-37c4-41f7-b1f3-b595e75b7664-kube-api-access-pvwv9\") pod \"limitador-limitador-54d6d988d5-gpvrw\" (UID: \"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:25.883528 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:25.883497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:26.016152 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.016123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:23:26.016866 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:23:26.016841    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9a7a8b_37c4_41f7_b1f3_b595e75b7664.slice/crio-8794674b29166df8660b84438151794ff8dacc3d0148b0abf6defff4c4e24cb2 WatchSource:0}: Error finding container 8794674b29166df8660b84438151794ff8dacc3d0148b0abf6defff4c4e24cb2: Status 404 returned error can't find the container with id 8794674b29166df8660b84438151794ff8dacc3d0148b0abf6defff4c4e24cb2
Jul 29 01:23:26.192538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.192463    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:23:26.196870 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.196854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:23:26.199694 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.199674    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bbhsp\""
Jul 29 01:23:26.202370 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.202348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:23:26.258577 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.258538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x26\" (UniqueName: \"kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26\") pod \"authorino-75d556bd77-l8lwg\" (UID: \"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3\") " pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:23:26.359117 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.359082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x26\" (UniqueName: \"kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26\") pod \"authorino-75d556bd77-l8lwg\" (UID: \"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3\") " pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:23:26.366933 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.366908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x26\" (UniqueName: \"kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26\") pod \"authorino-75d556bd77-l8lwg\" (UID: \"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3\") " pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:23:26.506634 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.506542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:23:26.537559 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.537337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw" event={"ID":"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664","Type":"ContainerStarted","Data":"8794674b29166df8660b84438151794ff8dacc3d0148b0abf6defff4c4e24cb2"}
Jul 29 01:23:26.685216 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:26.685148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:23:26.688993 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:23:26.688960    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod796a69ea_e1d4_4ef8_b08f_b72a8a48d9d3.slice/crio-4101ae07f7d2537c4aed8ed280ed14b8f658399f42d61ed6d55bdf14ae470562 WatchSource:0}: Error finding container 4101ae07f7d2537c4aed8ed280ed14b8f658399f42d61ed6d55bdf14ae470562: Status 404 returned error can't find the container with id 4101ae07f7d2537c4aed8ed280ed14b8f658399f42d61ed6d55bdf14ae470562
Jul 29 01:23:27.547600 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:27.547558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l8lwg" event={"ID":"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3","Type":"ContainerStarted","Data":"4101ae07f7d2537c4aed8ed280ed14b8f658399f42d61ed6d55bdf14ae470562"}
Jul 29 01:23:30.563906 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:30.563869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw" event={"ID":"8e9a7a8b-37c4-41f7-b1f3-b595e75b7664","Type":"ContainerStarted","Data":"ef997cd73e60897d0a47216773a051a2cca3435e93210dae62589554c500fd13"}
Jul 29 01:23:30.564356 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:30.563935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:30.565391 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:30.565369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l8lwg" event={"ID":"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3","Type":"ContainerStarted","Data":"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"}
Jul 29 01:23:30.581593 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:30.581547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw" podStartSLOduration=1.309419742 podStartE2EDuration="5.581531978s" totalImagesPullingTime="4.272112236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:23:30.579957164 +0000 UTC m=+797.733402540" watchObservedRunningTime="2026-07-29 01:23:30.581531978 +0000 UTC m=+797.734977393"
Jul 29 01:23:30.594278 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:30.594245    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-l8lwg" podStartSLOduration=0.984361463 podStartE2EDuration="4.594230282s" totalImagesPullingTime="3.609868819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:23:30.592871605 +0000 UTC m=+797.746316981" watchObservedRunningTime="2026-07-29 01:23:30.594230282 +0000 UTC m=+797.747675658"
Jul 29 01:23:41.570880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:41.570853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"
Jul 29 01:23:58.104717 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.104685    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:23:58.110604 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.110588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.113249 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.113222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 01:23:58.117837 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.117710    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:23:58.197936 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.197902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.198124 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.197964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnzz\" (UniqueName: \"kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.298832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.298795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnzz\" (UniqueName: \"kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.299004 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.298845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.301256 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.301226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.307578 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.307540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnzz\" (UniqueName: \"kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz\") pod \"authorino-7549664749-7gq8l\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") " pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.420530 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.420457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:23:58.543188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.543166    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:23:58.545530 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:23:58.545503    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6784ec46_f96f_4903_9248_d4712f144570.slice/crio-3f55f08a89429e617772fe112a9d73e40ae81520385eff12eba9c5881051958f WatchSource:0}: Error finding container 3f55f08a89429e617772fe112a9d73e40ae81520385eff12eba9c5881051958f: Status 404 returned error can't find the container with id 3f55f08a89429e617772fe112a9d73e40ae81520385eff12eba9c5881051958f
Jul 29 01:23:58.684110 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:58.684021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549664749-7gq8l" event={"ID":"6784ec46-f96f-4903-9248-d4712f144570","Type":"ContainerStarted","Data":"3f55f08a89429e617772fe112a9d73e40ae81520385eff12eba9c5881051958f"}
Jul 29 01:23:59.690019 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:59.689987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549664749-7gq8l" event={"ID":"6784ec46-f96f-4903-9248-d4712f144570","Type":"ContainerStarted","Data":"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"}
Jul 29 01:23:59.706880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:59.706839    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7549664749-7gq8l" podStartSLOduration=1.383685028 podStartE2EDuration="1.706827577s" totalImagesPullingTime="323.142549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:23:59.705012791 +0000 UTC m=+826.858458166" watchObservedRunningTime="2026-07-29 01:23:59.706827577 +0000 UTC m=+826.860272991"
Jul 29 01:23:59.734945 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:59.734921    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:23:59.735117 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:59.735098    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-l8lwg" podUID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" containerName="authorino" containerID="cri-o://797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f" gracePeriod=30
Jul 29 01:23:59.977897 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:23:59.977873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:24:00.014619 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.014592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9x26\" (UniqueName: \"kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26\") pod \"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3\" (UID: \"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3\") "
Jul 29 01:24:00.016644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.016619    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26" (OuterVolumeSpecName: "kube-api-access-k9x26") pod "796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" (UID: "796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3"). InnerVolumeSpecName "kube-api-access-k9x26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:24:00.115983 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.115937    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9x26\" (UniqueName: \"kubernetes.io/projected/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3-kube-api-access-k9x26\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:24:00.694930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.694895    2577 generic.go:356] "Generic (PLEG): container finished" podID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" containerID="797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f" exitCode=0
Jul 29 01:24:00.695409 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.694939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-l8lwg"
Jul 29 01:24:00.695409 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.694979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l8lwg" event={"ID":"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3","Type":"ContainerDied","Data":"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"}
Jul 29 01:24:00.695409 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.695029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-l8lwg" event={"ID":"796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3","Type":"ContainerDied","Data":"4101ae07f7d2537c4aed8ed280ed14b8f658399f42d61ed6d55bdf14ae470562"}
Jul 29 01:24:00.695409 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.695067    2577 scope.go:117] "RemoveContainer" containerID="797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"
Jul 29 01:24:00.703684 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.703667    2577 scope.go:117] "RemoveContainer" containerID="797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"
Jul 29 01:24:00.703936 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:24:00.703917    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f\": container with ID starting with 797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f not found: ID does not exist" containerID="797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"
Jul 29 01:24:00.703990 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.703950    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f"} err="failed to get container status \"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f\": rpc error: code = NotFound desc = could not find container \"797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f\": container with ID starting with 797dcd889aca0b96f8d073d2b1ca142754bf104a914a4f8550db2d23fd79e49f not found: ID does not exist"
Jul 29 01:24:00.739328 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.739299    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:24:00.743788 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:00.743769    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-l8lwg"]
Jul 29 01:24:01.472699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:01.472663    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" path="/var/lib/kubelet/pods/796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3/volumes"
Jul 29 01:24:40.924164 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:24:40.924132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:25:07.452561 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:25:07.452457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:25:28.754428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:25:28.754393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:25:39.354812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:25:39.354776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:25:56.842398 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:25:56.842364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:26:05.547660 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:05.547629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:26:11.662167 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.662135    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"]
Jul 29 01:26:11.662536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.662463    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" containerName="authorino"
Jul 29 01:26:11.662536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.662474    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" containerName="authorino"
Jul 29 01:26:11.662645 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.662547    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="796a69ea-e1d4-4ef8-b08f-b72a8a48d9d3" containerName="authorino"
Jul 29 01:26:11.665410 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.665394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.668402 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.668384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mpzrx\""
Jul 29 01:26:11.668507 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.668439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 01:26:11.669674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.669657    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 01:26:11.669764 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.669719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 01:26:11.675110 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.675089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"]
Jul 29 01:26:11.710608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.710701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e38965c0-3f59-493d-b3ec-66ece70a5706-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.710701 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458mr\" (UniqueName: \"kubernetes.io/projected/e38965c0-3f59-493d-b3ec-66ece70a5706-kube-api-access-458mr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.710774 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.710774 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.710774 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.710757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811742 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e38965c0-3f59-493d-b3ec-66ece70a5706-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.811932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.811913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458mr\" (UniqueName: \"kubernetes.io/projected/e38965c0-3f59-493d-b3ec-66ece70a5706-kube-api-access-458mr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.812170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.812150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.812246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.812181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.812246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.812221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.814184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.814168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e38965c0-3f59-493d-b3ec-66ece70a5706-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.814256 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.814241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e38965c0-3f59-493d-b3ec-66ece70a5706-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.820609 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.820576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-458mr\" (UniqueName: \"kubernetes.io/projected/e38965c0-3f59-493d-b3ec-66ece70a5706-kube-api-access-458mr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ldclj\" (UID: \"e38965c0-3f59-493d-b3ec-66ece70a5706\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:11.974543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:11.974484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:12.100330 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:12.100305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"]
Jul 29 01:26:12.102716 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:26:12.102690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode38965c0_3f59_493d_b3ec_66ece70a5706.slice/crio-67b8d59153e61714a00c5bcf3d36c66d3e74a39dfb0197e7fdb559c641fcdf61 WatchSource:0}: Error finding container 67b8d59153e61714a00c5bcf3d36c66d3e74a39dfb0197e7fdb559c641fcdf61: Status 404 returned error can't find the container with id 67b8d59153e61714a00c5bcf3d36c66d3e74a39dfb0197e7fdb559c641fcdf61
Jul 29 01:26:12.104078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:12.104062    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:26:12.270925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:12.270859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj" event={"ID":"e38965c0-3f59-493d-b3ec-66ece70a5706","Type":"ContainerStarted","Data":"67b8d59153e61714a00c5bcf3d36c66d3e74a39dfb0197e7fdb559c641fcdf61"}
Jul 29 01:26:13.277373 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:13.277330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj" event={"ID":"e38965c0-3f59-493d-b3ec-66ece70a5706","Type":"ContainerStarted","Data":"b5756adeae7bb97bebfa0598efa0f7d3024446a8932e14857b3d5738a457c3aa"}
Jul 29 01:26:13.298589 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:13.298543    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj" podStartSLOduration=1.286194548 podStartE2EDuration="2.298528083s" totalImagesPullingTime="1.012333535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:13.296582636 +0000 UTC m=+960.450028003" watchObservedRunningTime="2026-07-29 01:26:13.298528083 +0000 UTC m=+960.451973458"
Jul 29 01:26:14.949384 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:14.949348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:26:21.975151 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:21.975112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:21.975151 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:21.975157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:21.987329 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:21.987305    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:26:22.327324 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:26:22.327295    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ldclj"
Jul 29 01:27:05.829768 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:05.829734    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d64f5d668-jmpml"]
Jul 29 01:27:05.833226 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:05.833209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:05.840622 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:05.840445    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d64f5d668-jmpml"]
Jul 29 01:27:05.955136 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:05.955092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6lr\" (UniqueName: \"kubernetes.io/projected/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-kube-api-access-xb6lr\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:05.955306 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:05.955153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-tls-cert\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.055846 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.055814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-tls-cert\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.055994 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.055881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6lr\" (UniqueName: \"kubernetes.io/projected/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-kube-api-access-xb6lr\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.058217 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.058194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-tls-cert\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.071398 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.071371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6lr\" (UniqueName: \"kubernetes.io/projected/1d47d37d-5a0d-4350-8c85-fb088e8bfb33-kube-api-access-xb6lr\") pod \"authorino-7d64f5d668-jmpml\" (UID: \"1d47d37d-5a0d-4350-8c85-fb088e8bfb33\") " pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.143364 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.143297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d64f5d668-jmpml"
Jul 29 01:27:06.269387 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.269354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d64f5d668-jmpml"]
Jul 29 01:27:06.272771 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:27:06.272747    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d47d37d_5a0d_4350_8c85_fb088e8bfb33.slice/crio-404808a91282a0e65c9c91a336d841b065212c2053d6006d0877dcc4027a227a WatchSource:0}: Error finding container 404808a91282a0e65c9c91a336d841b065212c2053d6006d0877dcc4027a227a: Status 404 returned error can't find the container with id 404808a91282a0e65c9c91a336d841b065212c2053d6006d0877dcc4027a227a
Jul 29 01:27:06.505717 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:06.505642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d64f5d668-jmpml" event={"ID":"1d47d37d-5a0d-4350-8c85-fb088e8bfb33","Type":"ContainerStarted","Data":"404808a91282a0e65c9c91a336d841b065212c2053d6006d0877dcc4027a227a"}
Jul 29 01:27:07.514101 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.514029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d64f5d668-jmpml" event={"ID":"1d47d37d-5a0d-4350-8c85-fb088e8bfb33","Type":"ContainerStarted","Data":"740b141556d65ba72807dab4e26b18cca2df14a5feab5e5412a5700c75846fbf"}
Jul 29 01:27:07.534667 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.534602    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d64f5d668-jmpml" podStartSLOduration=2.032738517 podStartE2EDuration="2.534581116s" totalImagesPullingTime="501.842599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:27:07.531987589 +0000 UTC m=+1014.685432965" watchObservedRunningTime="2026-07-29 01:27:07.534581116 +0000 UTC m=+1014.688026494"
Jul 29 01:27:07.577065 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.576530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:27:07.577065 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.576744    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7549664749-7gq8l" podUID="6784ec46-f96f-4903-9248-d4712f144570" containerName="authorino" containerID="cri-o://de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce" gracePeriod=30
Jul 29 01:27:07.827891 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.827863    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:27:07.878664 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.878636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwnzz\" (UniqueName: \"kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz\") pod \"6784ec46-f96f-4903-9248-d4712f144570\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") "
Jul 29 01:27:07.878818 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.878675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert\") pod \"6784ec46-f96f-4903-9248-d4712f144570\" (UID: \"6784ec46-f96f-4903-9248-d4712f144570\") "
Jul 29 01:27:07.880781 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.880744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz" (OuterVolumeSpecName: "kube-api-access-lwnzz") pod "6784ec46-f96f-4903-9248-d4712f144570" (UID: "6784ec46-f96f-4903-9248-d4712f144570"). InnerVolumeSpecName "kube-api-access-lwnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:27:07.889351 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.889314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6784ec46-f96f-4903-9248-d4712f144570" (UID: "6784ec46-f96f-4903-9248-d4712f144570"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:27:07.979934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.979891    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwnzz\" (UniqueName: \"kubernetes.io/projected/6784ec46-f96f-4903-9248-d4712f144570-kube-api-access-lwnzz\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:27:07.979934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:07.979922    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6784ec46-f96f-4903-9248-d4712f144570-tls-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:27:08.521289 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.521202    2577 generic.go:356] "Generic (PLEG): container finished" podID="6784ec46-f96f-4903-9248-d4712f144570" containerID="de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce" exitCode=0
Jul 29 01:27:08.521289 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.521260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549664749-7gq8l"
Jul 29 01:27:08.521783 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.521288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549664749-7gq8l" event={"ID":"6784ec46-f96f-4903-9248-d4712f144570","Type":"ContainerDied","Data":"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"}
Jul 29 01:27:08.521783 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.521332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549664749-7gq8l" event={"ID":"6784ec46-f96f-4903-9248-d4712f144570","Type":"ContainerDied","Data":"3f55f08a89429e617772fe112a9d73e40ae81520385eff12eba9c5881051958f"}
Jul 29 01:27:08.521783 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.521351    2577 scope.go:117] "RemoveContainer" containerID="de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"
Jul 29 01:27:08.530633 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.530615    2577 scope.go:117] "RemoveContainer" containerID="de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"
Jul 29 01:27:08.530853 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:27:08.530838    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce\": container with ID starting with de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce not found: ID does not exist" containerID="de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"
Jul 29 01:27:08.530917 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.530859    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce"} err="failed to get container status \"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce\": rpc error: code = NotFound desc = could not find container \"de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce\": container with ID starting with de970b5b65ea495297fbd6cc2721902afde83d82c8619d7717ab6fa2fe83c5ce not found: ID does not exist"
Jul 29 01:27:08.545138 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.545114    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:27:08.549610 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:08.549589    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7549664749-7gq8l"]
Jul 29 01:27:09.466250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:09.466213    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6784ec46-f96f-4903-9248-d4712f144570" path="/var/lib/kubelet/pods/6784ec46-f96f-4903-9248-d4712f144570/volumes"
Jul 29 01:27:35.659452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:35.659421    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:27:44.669651 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:44.669619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:27:54.552748 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:27:54.552711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:29:40.157474 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:29:40.157437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:29:51.760932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:29:51.760898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:30:26.651171 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:30:26.651135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:30:36.253475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:30:36.253442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:30:45.862546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:30:45.862503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:30:55.760425 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:30:55.760385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:31:05.553330 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:31:05.553249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:31:21.063546 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:31:21.063504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:31:28.555076 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:31:28.555018    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:31:36.353763 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:31:36.353731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:31:58.361839 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:31:58.361801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:32:08.354991 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:32:08.354954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:32:31.865816 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:32:31.865778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:32:42.557499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:32:42.557458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:32:52.861082 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:32:52.861034    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:08.557098 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:08.557069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:17.662797 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:17.662746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:28.051481 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:28.051441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:37.249078 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:37.249033    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:46.156575 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:46.156542    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:33:56.556931 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:33:56.556891    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:34:06.453711 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:34:06.453614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:35:47.402300 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:35:47.402217    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:35:57.865265 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:35:57.865234    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:36:11.798806 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:36:11.798768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:36:23.097967 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:36:23.097933    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:36:34.775327 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:36:34.775290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:36:43.424934 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:36:43.424893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:36:53.658542 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:36:53.658497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:37:03.960602 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:37:03.960567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:37:14.450006 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:37:14.449920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:37:24.753356 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:37:24.753322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:37:35.153248 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:37:35.153215    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:37:41.357378 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:37:41.357339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:38:04.107416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:04.107384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:38:13.657771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:13.657734    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:38:21.663823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.663792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:38:21.664185 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.664109    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6784ec46-f96f-4903-9248-d4712f144570" containerName="authorino"
Jul 29 01:38:21.664185 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.664120    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6784ec46-f96f-4903-9248-d4712f144570" containerName="authorino"
Jul 29 01:38:21.664185 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.664185    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6784ec46-f96f-4903-9248-d4712f144570" containerName="authorino"
Jul 29 01:38:21.666897 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.666882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.671109 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.671088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c99ce2d7-kserve-self-signed-certs\""
Jul 29 01:38:21.692214 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.692179    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:38:21.785720 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtpz\" (UniqueName: \"kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.785828 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.785828 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.785828 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.785950 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.785950 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.785845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.886977 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.886949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887286 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtpz\" (UniqueName: \"kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887286 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887437 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887437 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.887545 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.887521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.889176 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.889160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.889586 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.889566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.896034 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.896004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtpz\" (UniqueName: \"kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz\") pod \"e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") " pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:21.975352 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:21.975289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:22.098107 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:22.098085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:38:22.099960 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:38:22.099933    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4582bd04_7793_4fbc_89c8_ca5dc79be0ca.slice/crio-a9e085caf4c899047080b5bba2497e985dc97534b358aaa866d750cb748b3272 WatchSource:0}: Error finding container a9e085caf4c899047080b5bba2497e985dc97534b358aaa866d750cb748b3272: Status 404 returned error can't find the container with id a9e085caf4c899047080b5bba2497e985dc97534b358aaa866d750cb748b3272
Jul 29 01:38:22.455739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:22.455704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" event={"ID":"4582bd04-7793-4fbc-89c8-ca5dc79be0ca","Type":"ContainerStarted","Data":"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"}
Jul 29 01:38:22.455916 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:22.455750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" event={"ID":"4582bd04-7793-4fbc-89c8-ca5dc79be0ca","Type":"ContainerStarted","Data":"a9e085caf4c899047080b5bba2497e985dc97534b358aaa866d750cb748b3272"}
Jul 29 01:38:22.472683 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:22.472643    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" podStartSLOduration=1.4726295 podStartE2EDuration="1.4726295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:38:22.471537805 +0000 UTC m=+1689.624983181" watchObservedRunningTime="2026-07-29 01:38:22.4726295 +0000 UTC m=+1689.626074875"
Jul 29 01:38:25.457433 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:25.457401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:38:31.748397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.748365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:38:31.752065 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.752029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.755092 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.755069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c99ce2d7-kserve-self-signed-certs\""
Jul 29 01:38:31.760642 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.760621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:38:31.866550 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.866672 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.866725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.866725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.866818 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbn2\" (UniqueName: \"kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.866818 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.866751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967899 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967899 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967899 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbn2\" (UniqueName: \"kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.967899 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.967824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.968177 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.968156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.968239 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.968163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.968277 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.968259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.970049 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.970014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.970170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.970154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.975414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.975395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:31.975527 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.975432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:31.976792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.976771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbn2\" (UniqueName: \"kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2\") pod \"e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") " pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:31.988892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:31.988875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:32.062410 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.062386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:32.188840 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.188813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:38:32.191699 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:38:32.191677    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38ebf8bf_32b3_4d93_8c72_70c40c618672.slice/crio-ac11b864435536aae9360b08b210e88f86669b4e790f75d486fc7ddb4b596e7a WatchSource:0}: Error finding container ac11b864435536aae9360b08b210e88f86669b4e790f75d486fc7ddb4b596e7a: Status 404 returned error can't find the container with id ac11b864435536aae9360b08b210e88f86669b4e790f75d486fc7ddb4b596e7a
Jul 29 01:38:32.498922 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.498837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" event={"ID":"38ebf8bf-32b3-4d93-8c72-70c40c618672","Type":"ContainerStarted","Data":"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"}
Jul 29 01:38:32.498922 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.498877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" event={"ID":"38ebf8bf-32b3-4d93-8c72-70c40c618672","Type":"ContainerStarted","Data":"ac11b864435536aae9360b08b210e88f86669b4e790f75d486fc7ddb4b596e7a"}
Jul 29 01:38:32.510180 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.510159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:38:32.518575 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:32.518535    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" podStartSLOduration=1.518521426 podStartE2EDuration="1.518521426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:38:32.516500164 +0000 UTC m=+1699.669945541" watchObservedRunningTime="2026-07-29 01:38:32.518521426 +0000 UTC m=+1699.671966802"
Jul 29 01:38:42.063007 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:42.062975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:42.063411 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:42.063029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:42.074807 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:42.074786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:42.556162 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:42.556091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:38:42.951182 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:42.951150    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:38:58.981167 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:38:58.981133    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:39:00.066555 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:00.066525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:39:03.153966 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:03.153935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:39:14.000160 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.000091    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" podUID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" containerName="main" containerID="cri-o://f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe" gracePeriod=45
Jul 29 01:39:14.242205 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.242180    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:39:14.398975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.398951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.398990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399008    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdtpz\" (UniqueName: \"kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399123 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm\") pod \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\" (UID: \"4582bd04-7793-4fbc-89c8-ca5dc79be0ca\") "
Jul 29 01:39:14.399426 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:14.399426 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:14.399426 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.399336    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home" (OuterVolumeSpecName: "home") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:14.401188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.401161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm" (OuterVolumeSpecName: "dshm") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:14.401277 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.401207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:39:14.401277 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.401234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz" (OuterVolumeSpecName: "kube-api-access-bdtpz") pod "4582bd04-7793-4fbc-89c8-ca5dc79be0ca" (UID: "4582bd04-7793-4fbc-89c8-ca5dc79be0ca"). InnerVolumeSpecName "kube-api-access-bdtpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:39:14.500297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500270    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdtpz\" (UniqueName: \"kubernetes.io/projected/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-kube-api-access-bdtpz\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.500297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500297    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.500297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500306    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.500457 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500313    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.500457 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500322    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.500457 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.500329    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4582bd04-7793-4fbc-89c8-ca5dc79be0ca-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:14.690343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.690282    2577 generic.go:356] "Generic (PLEG): container finished" podID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" containerID="f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe" exitCode=0
Jul 29 01:39:14.690343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.690325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"
Jul 29 01:39:14.690488 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.690357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" event={"ID":"4582bd04-7793-4fbc-89c8-ca5dc79be0ca","Type":"ContainerDied","Data":"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"}
Jul 29 01:39:14.690488 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.690387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw" event={"ID":"4582bd04-7793-4fbc-89c8-ca5dc79be0ca","Type":"ContainerDied","Data":"a9e085caf4c899047080b5bba2497e985dc97534b358aaa866d750cb748b3272"}
Jul 29 01:39:14.690488 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.690407    2577 scope.go:117] "RemoveContainer" containerID="f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"
Jul 29 01:39:14.702258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.699922    2577 scope.go:117] "RemoveContainer" containerID="f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"
Jul 29 01:39:14.702613 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:39:14.702431    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe\": container with ID starting with f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe not found: ID does not exist" containerID="f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"
Jul 29 01:39:14.702703 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.702621    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe"} err="failed to get container status \"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe\": rpc error: code = NotFound desc = could not find container \"f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe\": container with ID starting with f3f5100e4b7257ff32177696d91daa464082ba69871bb141a5dd28997ff454fe not found: ID does not exist"
Jul 29 01:39:14.716698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.716672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:39:14.720719 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:14.720699    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c99ce2d7-kserve-6749776495-vdkfw"]
Jul 29 01:39:15.085097 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.085049    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" podUID="38ebf8bf-32b3-4d93-8c72-70c40c618672" containerName="main" containerID="cri-o://c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f" gracePeriod=45
Jul 29 01:39:15.328478 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.328454    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:39:15.407239 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407239 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqbn2\" (UniqueName: \"kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache\") pod \"38ebf8bf-32b3-4d93-8c72-70c40c618672\" (UID: \"38ebf8bf-32b3-4d93-8c72-70c40c618672\") "
Jul 29 01:39:15.407611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407559    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:15.407689 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home" (OuterVolumeSpecName: "home") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:15.407750 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.407711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache" (OuterVolumeSpecName: "model-cache") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:15.409279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.409253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2" (OuterVolumeSpecName: "kube-api-access-kqbn2") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "kube-api-access-kqbn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:39:15.409279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.409273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:39:15.409408 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.409258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm" (OuterVolumeSpecName: "dshm") pod "38ebf8bf-32b3-4d93-8c72-70c40c618672" (UID: "38ebf8bf-32b3-4d93-8c72-70c40c618672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:39:15.466717 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.466693    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" path="/var/lib/kubelet/pods/4582bd04-7793-4fbc-89c8-ca5dc79be0ca/volumes"
Jul 29 01:39:15.508644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508620    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ebf8bf-32b3-4d93-8c72-70c40c618672-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.508644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508641    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbn2\" (UniqueName: \"kubernetes.io/projected/38ebf8bf-32b3-4d93-8c72-70c40c618672-kube-api-access-kqbn2\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.508761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508651    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.508761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508660    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.508761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508668    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.508761 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.508675    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ebf8bf-32b3-4d93-8c72-70c40c618672-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:39:15.697462 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.697384    2577 generic.go:356] "Generic (PLEG): container finished" podID="38ebf8bf-32b3-4d93-8c72-70c40c618672" containerID="c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f" exitCode=0
Jul 29 01:39:15.697462 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.697425    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"
Jul 29 01:39:15.697699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.697463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" event={"ID":"38ebf8bf-32b3-4d93-8c72-70c40c618672","Type":"ContainerDied","Data":"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"}
Jul 29 01:39:15.697699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.697500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7" event={"ID":"38ebf8bf-32b3-4d93-8c72-70c40c618672","Type":"ContainerDied","Data":"ac11b864435536aae9360b08b210e88f86669b4e790f75d486fc7ddb4b596e7a"}
Jul 29 01:39:15.697699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.697523    2577 scope.go:117] "RemoveContainer" containerID="c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"
Jul 29 01:39:15.705710 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.705691    2577 scope.go:117] "RemoveContainer" containerID="c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"
Jul 29 01:39:15.705971 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:39:15.705950    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f\": container with ID starting with c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f not found: ID does not exist" containerID="c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"
Jul 29 01:39:15.706010 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.705980    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f"} err="failed to get container status \"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f\": rpc error: code = NotFound desc = could not find container \"c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f\": container with ID starting with c7dc21a4865a1517ad6280a94c658e534d295d1f4d8497bbd4d391820a5c819f not found: ID does not exist"
Jul 29 01:39:15.713183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.713161    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:39:15.718033 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:15.718012    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c99ce2d7-kserve-9b4688c9f-kk2g7"]
Jul 29 01:39:16.855254 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:16.855221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:39:17.466281 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:17.466249    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ebf8bf-32b3-4d93-8c72-70c40c618672" path="/var/lib/kubelet/pods/38ebf8bf-32b3-4d93-8c72-70c40c618672/volumes"
Jul 29 01:39:23.272383 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:23.272349    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:39:37.350000 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:37.349956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:39:47.860542 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:47.860505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:39:56.551412 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:39:56.551378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:06.552794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:06.552712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:15.556952 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:15.556909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:25.651074 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:25.651029    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:38.760582 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:38.760541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:47.667058 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:47.667010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:40:57.453053 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:40:57.453010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:06.657095 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:06.657061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:18.554245 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:18.554209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:26.157196 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:26.157155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:36.860005 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:36.859920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:48.450339 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:48.450306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:41:58.655669 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:41:58.655635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:42:06.357020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:42:06.356983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:42:20.151227 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:42:20.151191    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:42:30.754887 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:42:30.754851    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:42:40.574547 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:42:40.574511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:42:52.553028 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:42:52.552995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:43:08.041600 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041525    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"]
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041844    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ebf8bf-32b3-4d93-8c72-70c40c618672" containerName="main"
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041855    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ebf8bf-32b3-4d93-8c72-70c40c618672" containerName="main"
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041877    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" containerName="main"
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041882    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" containerName="main"
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041964    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ebf8bf-32b3-4d93-8c72-70c40c618672" containerName="main"
Jul 29 01:43:08.042014 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.041977    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4582bd04-7793-4fbc-89c8-ca5dc79be0ca" containerName="main"
Jul 29 01:43:08.044965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.044949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:08.047515 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.047498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 01:43:08.047583 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.047558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:43:08.051923 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.051896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"]
Jul 29 01:43:08.214580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.214551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb\" (UID: \"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:08.315287 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.315260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb\" (UID: \"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:08.315818 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.315800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb\" (UID: \"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:08.356396 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.356378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:08.494086 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.494059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"]
Jul 29 01:43:08.496657 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:43:08.496632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb817ba5e_0c9d_4202_ac8e_0fc7755e2dd9.slice/crio-0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac WatchSource:0}: Error finding container 0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac: Status 404 returned error can't find the container with id 0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac
Jul 29 01:43:08.498849 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.498834    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:43:08.714147 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:08.714068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb" event={"ID":"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9","Type":"ContainerStarted","Data":"0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac"}
Jul 29 01:43:11.729402 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:11.729316    2577 generic.go:356] "Generic (PLEG): container finished" podID="b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" containerID="e23530e8729a28e086bd994db29827277126b2721867494a5ded483126cd8dc7" exitCode=0
Jul 29 01:43:11.729746 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:11.729388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb" event={"ID":"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9","Type":"ContainerDied","Data":"e23530e8729a28e086bd994db29827277126b2721867494a5ded483126cd8dc7"}
Jul 29 01:43:12.856899 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:12.856876    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:13.054913 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.054827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca\") pod \"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9\" (UID: \"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9\") "
Jul 29 01:43:13.055203 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.055181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" (UID: "b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:43:13.155555 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.155522    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:43:13.738875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.738786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb" event={"ID":"b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9","Type":"ContainerDied","Data":"0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac"}
Jul 29 01:43:13.738875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.738842    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0652cc0de39c93f7c612780b404ed9ccaf084adbe485b09c00600613769b51ac"
Jul 29 01:43:13.738875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:13.738806    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"
Jul 29 01:43:23.414862 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:23.414833    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"]
Jul 29 01:43:23.418516 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:23.418489    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9d88784-4wxvb"]
Jul 29 01:43:23.467055 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:23.467019    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" path="/var/lib/kubelet/pods/b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9/volumes"
Jul 29 01:43:24.567112 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.567079    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:43:24.567477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.567405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" containerName="revoke-keys"
Jul 29 01:43:24.567477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.567417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" containerName="revoke-keys"
Jul 29 01:43:24.567547 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.567494    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b817ba5e-0c9d-4202-ac8e-0fc7755e2dd9" containerName="revoke-keys"
Jul 29 01:43:24.573237 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.573218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.575643 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.575623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 01:43:24.575764 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.575648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a63cc33b-gw-openshift-default-dockercfg-vxx56\""
Jul 29 01:43:24.581903 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.581882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:43:24.641341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqnj8\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641580 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.641669 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.641578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742862 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.742989 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.742976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqnj8\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743219 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743219 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743219 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743440 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743508 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743564 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743602 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.743855 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.743834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.745259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.745231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.745515 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.745494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.758414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.758393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqnj8\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.758797 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.758775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token\") pod \"e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") " pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:24.884473 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:24.884402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:25.015355 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.015300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:43:25.017962 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:43:25.017938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09152dc0_8b67_4941_b547_99dc02dc77ab.slice/crio-9e5513228fd1f1f88be05add2916e5f178fb12921f5573bd08c8f6ecc26e7032 WatchSource:0}: Error finding container 9e5513228fd1f1f88be05add2916e5f178fb12921f5573bd08c8f6ecc26e7032: Status 404 returned error can't find the container with id 9e5513228fd1f1f88be05add2916e5f178fb12921f5573bd08c8f6ecc26e7032
Jul 29 01:43:25.020114 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.020084    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:25.020200 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.020146    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:25.020200 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.020177    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:25.791830 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.791797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerStarted","Data":"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"}
Jul 29 01:43:25.792205 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.791845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerStarted","Data":"9e5513228fd1f1f88be05add2916e5f178fb12921f5573bd08c8f6ecc26e7032"}
Jul 29 01:43:25.884503 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:25.884474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:26.890051 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:26.890006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:26.911530 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:26.911482    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podStartSLOduration=2.911468781 podStartE2EDuration="2.911468781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:43:25.812617149 +0000 UTC m=+1992.966062525" watchObservedRunningTime="2026-07-29 01:43:26.911468781 +0000 UTC m=+1994.064914157"
Jul 29 01:43:27.153273 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:27.153203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:43:27.799799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:27.799761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:27.800592 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:27.800574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:37.762095 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:37.762060    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:43:41.866626 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.866589    2577 generic.go:356] "Generic (PLEG): container finished" podID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerID="9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b" exitCode=0
Jul 29 01:43:41.867031 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.866663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerDied","Data":"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"}
Jul 29 01:43:41.867031 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.867019    2577 scope.go:117] "RemoveContainer" containerID="9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"
Jul 29 01:43:41.867986 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.867957    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:41.868113 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.868077    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:41.868178 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:41.868129    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:43:42.800460 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:42.800427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:42.872970 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:42.872929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerStarted","Data":"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"}
Jul 29 01:43:42.885173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:42.885150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:42.886239 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:42.886219    2577 patch_prober.go:28] interesting pod/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 29 01:43:42.886291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:42.886270    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 29 01:43:43.885020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:43.884984    2577 patch_prober.go:28] interesting pod/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 29 01:43:43.885412 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:43.885031    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 29 01:43:44.885100 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:44.885065    2577 patch_prober.go:28] interesting pod/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 29 01:43:44.885477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:44.885114    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 29 01:43:45.884712 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:45.884672    2577 patch_prober.go:28] interesting pod/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 29 01:43:45.884882 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:45.884734    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 29 01:43:46.889193 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:46.889160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:46.889597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:46.889448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:43:46.890214 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:43:46.890190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:44:08.536902 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:08.536860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:44:08.537466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:08.537091    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy" containerID="cri-o://b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50" gracePeriod=30
Jul 29 01:44:12.356146 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:12.356110    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:44:13.784105 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.784075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:44:13.940588 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940588 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqnj8\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.940803 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data\") pod \"09152dc0-8b67-4941-b547-99dc02dc77ab\" (UID: \"09152dc0-8b67-4941-b547-99dc02dc77ab\") "
Jul 29 01:44:13.941183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.940919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:44:13.941183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.941015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:44:13.941183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.941022    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:44:13.941183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.941087    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09152dc0-8b67-4941-b547-99dc02dc77ab-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:13.941383 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.941317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:44:13.941723 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.941702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:44:13.943021 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.942996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:44:13.943021 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.943013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:44:13.943276 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.943210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 01:44:13.943276 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:13.943213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8" (OuterVolumeSpecName: "kube-api-access-xqnj8") pod "09152dc0-8b67-4941-b547-99dc02dc77ab" (UID: "09152dc0-8b67-4941-b547-99dc02dc77ab"). InnerVolumeSpecName "kube-api-access-xqnj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:44:14.010412 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.010382    2577 generic.go:356] "Generic (PLEG): container finished" podID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerID="b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50" exitCode=0
Jul 29 01:44:14.010504 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.010430    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"
Jul 29 01:44:14.010504 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.010463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerDied","Data":"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"}
Jul 29 01:44:14.010575 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.010511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf" event={"ID":"09152dc0-8b67-4941-b547-99dc02dc77ab","Type":"ContainerDied","Data":"9e5513228fd1f1f88be05add2916e5f178fb12921f5573bd08c8f6ecc26e7032"}
Jul 29 01:44:14.010575 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.010532    2577 scope.go:117] "RemoveContainer" containerID="b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"
Jul 29 01:44:14.019150 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.019132    2577 scope.go:117] "RemoveContainer" containerID="9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"
Jul 29 01:44:14.026544 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.026524    2577 scope.go:117] "RemoveContainer" containerID="b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"
Jul 29 01:44:14.026779 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:44:14.026765    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50\": container with ID starting with b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50 not found: ID does not exist" containerID="b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"
Jul 29 01:44:14.026841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.026785    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50"} err="failed to get container status \"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50\": rpc error: code = NotFound desc = could not find container \"b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50\": container with ID starting with b44c98cd88850277f92eb4fda01c80c595b88b15cbdff4338eeeb705ee4b1b50 not found: ID does not exist"
Jul 29 01:44:14.026841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.026800    2577 scope.go:117] "RemoveContainer" containerID="9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"
Jul 29 01:44:14.027060 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:44:14.027020    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b\": container with ID starting with 9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b not found: ID does not exist" containerID="9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"
Jul 29 01:44:14.027128 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.027062    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b"} err="failed to get container status \"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b\": rpc error: code = NotFound desc = could not find container \"9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b\": container with ID starting with 9d1d6ab816d983bb9877dfbad7374390780e3ac3002cd5fe9b38c9ef5ac3878b not found: ID does not exist"
Jul 29 01:44:14.034007 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.033985    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:44:14.037882 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.037862    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a63cc33b-gw-openshift-default-58665db596-cjjwf"]
Jul 29 01:44:14.041579 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041561    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041581    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041591    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041598    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041606    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041614    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041622    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09152dc0-8b67-4941-b547-99dc02dc77ab-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:14.041644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:14.041630    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqnj8\" (UniqueName: \"kubernetes.io/projected/09152dc0-8b67-4941-b547-99dc02dc77ab-kube-api-access-xqnj8\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:15.466457 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:15.466423    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" path="/var/lib/kubelet/pods/09152dc0-8b67-4941-b547-99dc02dc77ab/volumes"
Jul 29 01:44:23.357263 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:23.357229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:44:37.710941 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.710908    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"]
Jul 29 01:44:37.711313 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.711229    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:37.711313 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.711240    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:37.711394 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.711317    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:37.714268 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.714252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:37.717130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.717107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 01:44:37.717130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.717122    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:44:37.722278 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.722255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"]
Jul 29 01:44:37.821751 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.821729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0187de68-rgqsj\" (UID: \"1f04e0d9-e355-4c74-98f7-41d1111f48df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:37.922646 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.922608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0187de68-rgqsj\" (UID: \"1f04e0d9-e355-4c74-98f7-41d1111f48df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:37.923200 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:37.923183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0187de68-rgqsj\" (UID: \"1f04e0d9-e355-4c74-98f7-41d1111f48df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:38.025865 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:38.025791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:38.152415 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:38.152391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"]
Jul 29 01:44:38.154668 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:44:38.154643    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f04e0d9_e355_4c74_98f7_41d1111f48df.slice/crio-abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744 WatchSource:0}: Error finding container abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744: Status 404 returned error can't find the container with id abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744
Jul 29 01:44:39.129129 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:39.129101    2577 generic.go:356] "Generic (PLEG): container finished" podID="1f04e0d9-e355-4c74-98f7-41d1111f48df" containerID="b98b94997207eb5ad1c0f383b3f294cefcdf02e8720083a3d71cc571d42fc96e" exitCode=0
Jul 29 01:44:39.129458 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:39.129191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj" event={"ID":"1f04e0d9-e355-4c74-98f7-41d1111f48df","Type":"ContainerDied","Data":"b98b94997207eb5ad1c0f383b3f294cefcdf02e8720083a3d71cc571d42fc96e"}
Jul 29 01:44:39.129458 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:39.129241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj" event={"ID":"1f04e0d9-e355-4c74-98f7-41d1111f48df","Type":"ContainerStarted","Data":"abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744"}
Jul 29 01:44:40.258698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:40.258669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:40.343082 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:40.343061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca\") pod \"1f04e0d9-e355-4c74-98f7-41d1111f48df\" (UID: \"1f04e0d9-e355-4c74-98f7-41d1111f48df\") "
Jul 29 01:44:40.343367 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:40.343347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f04e0d9-e355-4c74-98f7-41d1111f48df" (UID: "1f04e0d9-e355-4c74-98f7-41d1111f48df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:44:40.443904 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:40.443845    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f04e0d9-e355-4c74-98f7-41d1111f48df-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:44:41.139375 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:41.139342    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"
Jul 29 01:44:41.139375 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:41.139364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj" event={"ID":"1f04e0d9-e355-4c74-98f7-41d1111f48df","Type":"ContainerDied","Data":"abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744"}
Jul 29 01:44:41.139572 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:41.139432    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf0239aa464236e14ae2db9b2b36a6487b9d8317c9b01ebba5cb78fba26e744"
Jul 29 01:44:43.085988 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:43.085956    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"]
Jul 29 01:44:43.092665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:43.092640    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0187de68-rgqsj"]
Jul 29 01:44:43.466500 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:43.466432    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f04e0d9-e355-4c74-98f7-41d1111f48df" path="/var/lib/kubelet/pods/1f04e0d9-e355-4c74-98f7-41d1111f48df/volumes"
Jul 29 01:44:45.123191 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123159    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:44:45.123551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123488    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f04e0d9-e355-4c74-98f7-41d1111f48df" containerName="revoke-keys"
Jul 29 01:44:45.123551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123500    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f04e0d9-e355-4c74-98f7-41d1111f48df" containerName="revoke-keys"
Jul 29 01:44:45.123551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123514    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:45.123551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123519    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:45.123672 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123580    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09152dc0-8b67-4941-b547-99dc02dc77ab" containerName="istio-proxy"
Jul 29 01:44:45.123672 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.123595    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f04e0d9-e355-4c74-98f7-41d1111f48df" containerName="revoke-keys"
Jul 29 01:44:45.126708 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.126692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.129568 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.129544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ff6a497a-openshift-default-dockercfg-2dhmt\""
Jul 29 01:44:45.129685 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.129632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 01:44:45.158455 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.158436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:44:45.183105 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183193 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183193 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183272 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183272 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183272 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6px7n\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.183416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.183369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284347 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6px7n\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284477 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284721 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284721 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284721 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284879 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.284879 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.285004 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.284974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.285132 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.285110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.285285 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.285265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.286593 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.286567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.286792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.286775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.306433 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.306417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.320896 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.320874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6px7n\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n\") pod \"e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") " pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.437082 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.436986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:45.467855 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.467826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:44:45.568760 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.568727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:44:45.571778 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:44:45.571751    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcaa63f3_c5d4_43c3_ba4e_ca6f1405eba1.slice/crio-a19862fa4bdf8f44032cd585bb86dcda9034a227bef871926533a88ba566cfc3 WatchSource:0}: Error finding container a19862fa4bdf8f44032cd585bb86dcda9034a227bef871926533a88ba566cfc3: Status 404 returned error can't find the container with id a19862fa4bdf8f44032cd585bb86dcda9034a227bef871926533a88ba566cfc3
Jul 29 01:44:45.573775 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.573741    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:44:45.573858 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.573818    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:44:45.573924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:45.573861    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:44:46.162122 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:46.162082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerStarted","Data":"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"}
Jul 29 01:44:46.162489 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:46.162175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerStarted","Data":"a19862fa4bdf8f44032cd585bb86dcda9034a227bef871926533a88ba566cfc3"}
Jul 29 01:44:46.195771 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:46.195719    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podStartSLOduration=1.195705727 podStartE2EDuration="1.195705727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:44:46.192517271 +0000 UTC m=+2073.345962647" watchObservedRunningTime="2026-07-29 01:44:46.195705727 +0000 UTC m=+2073.349151102"
Jul 29 01:44:46.437556 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:46.437474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:46.442388 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:46.442365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:47.166858 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:47.166822    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:47.167787 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:47.167766    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:44:56.775235 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:44:56.775202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:45:01.231667 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.231629    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerID="0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd" exitCode=0
Jul 29 01:45:01.232119 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.231704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerDied","Data":"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"}
Jul 29 01:45:01.232119 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.232085    2577 scope.go:117] "RemoveContainer" containerID="0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"
Jul 29 01:45:01.233013 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.232988    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:45:01.233140 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.233080    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:45:01.233208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:01.233140    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:45:02.167252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:02.167206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:02.240113 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:02.240073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerStarted","Data":"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"}
Jul 29 01:45:02.437298 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:02.437186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:02.438536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:02.438506    2577 patch_prober.go:28] interesting pod/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 29 01:45:02.438756 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:02.438730    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 29 01:45:03.438103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:03.438067    2577 patch_prober.go:28] interesting pod/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 29 01:45:03.438571 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:03.438133    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 29 01:45:04.438281 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:04.438243    2577 patch_prober.go:28] interesting pod/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 29 01:45:04.438773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:04.438312    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 29 01:45:05.438273 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:05.438238    2577 patch_prober.go:28] interesting pod/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 29 01:45:05.438538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:05.438507    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 29 01:45:06.441698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:06.441669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:06.442135 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:06.441931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:06.442515 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:06.442494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:29.077762 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:29.077726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:45:29.078354 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:29.077955    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy" containerID="cri-o://c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e" gracePeriod=30
Jul 29 01:45:32.264276 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:32.264241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:45:34.322928 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.322904    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:34.368611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6px7n\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368611 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368759    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token\") pod \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\" (UID: \"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1\") "
Jul 29 01:45:34.368875 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:45:34.369279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.368931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:45:34.369279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.369140    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.369279 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.369160    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.369444 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.369420    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:45:34.369502 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.369451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:45:34.370151 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.370118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:45:34.371256 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.371235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:45:34.371256 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.371247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:45:34.371466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.371247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n" (OuterVolumeSpecName: "kube-api-access-6px7n") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "kube-api-access-6px7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:45:34.371656 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.371639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" (UID: "bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 01:45:34.394198 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.394175    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerID="c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e" exitCode=0
Jul 29 01:45:34.394284 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.394218    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"
Jul 29 01:45:34.394344 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.394219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerDied","Data":"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"}
Jul 29 01:45:34.394400 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.394368    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl" event={"ID":"bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1","Type":"ContainerDied","Data":"a19862fa4bdf8f44032cd585bb86dcda9034a227bef871926533a88ba566cfc3"}
Jul 29 01:45:34.394452 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.394411    2577 scope.go:117] "RemoveContainer" containerID="c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"
Jul 29 01:45:34.402897 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.402881    2577 scope.go:117] "RemoveContainer" containerID="0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"
Jul 29 01:45:34.410359 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.410347    2577 scope.go:117] "RemoveContainer" containerID="c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"
Jul 29 01:45:34.410609 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:45:34.410590    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e\": container with ID starting with c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e not found: ID does not exist" containerID="c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"
Jul 29 01:45:34.410673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.410615    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e"} err="failed to get container status \"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e\": rpc error: code = NotFound desc = could not find container \"c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e\": container with ID starting with c9d6f73186d22e4f3f98ff8feddc811e442e7f8d6647b1624a33078b47b2f07e not found: ID does not exist"
Jul 29 01:45:34.410673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.410631    2577 scope.go:117] "RemoveContainer" containerID="0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"
Jul 29 01:45:34.410858 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:45:34.410841    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd\": container with ID starting with 0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd not found: ID does not exist" containerID="0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"
Jul 29 01:45:34.410898 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.410861    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd"} err="failed to get container status \"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd\": rpc error: code = NotFound desc = could not find container \"0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd\": container with ID starting with 0c3201ba955d64459c4b1f55fa29f1a61404cdb5b62e27ed05845f0c9090e2cd not found: ID does not exist"
Jul 29 01:45:34.416745 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.416726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:45:34.420920 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.420900    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ff6a497a-openshift-default-7575b5d944-64ppl"]
Jul 29 01:45:34.469809 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469786    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6px7n\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-kube-api-access-6px7n\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469809 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469807    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469817    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469826    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469835    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469842    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:34.469935 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:34.469850    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:45:35.466436 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:35.466402    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" path="/var/lib/kubelet/pods/bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1/volumes"
Jul 29 01:45:42.380539 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:45:42.380507    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:46:54.357522 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:46:54.357490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:47:05.988924 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:05.988889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:47:15.654873 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:15.654834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:47:31.858861 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:31.858817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:47:54.883993 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.883958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"]
Jul 29 01:47:54.884434 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.884287    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:47:54.884434 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.884298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:47:54.884434 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.884374    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:47:54.887231 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.887215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:54.893894 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.893871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:47:54.894259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.894241    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 01:47:54.965620 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.965590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"]
Jul 29 01:47:54.994815 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:54.994789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh\" (UID: \"6cbc34cd-60de-4f93-8f19-1399f6ba047c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:55.095760 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:55.095734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh\" (UID: \"6cbc34cd-60de-4f93-8f19-1399f6ba047c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:55.096380 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:55.096358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh\" (UID: \"6cbc34cd-60de-4f93-8f19-1399f6ba047c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:55.196791 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:55.196722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:55.320216 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:55.320181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"]
Jul 29 01:47:55.323057 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:47:55.323019    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbc34cd_60de_4f93_8f19_1399f6ba047c.slice/crio-ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f WatchSource:0}: Error finding container ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f: Status 404 returned error can't find the container with id ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f
Jul 29 01:47:56.020795 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:56.020767    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cbc34cd-60de-4f93-8f19-1399f6ba047c" containerID="7a1b2f30adc25b627d9e2f8da4783e43f631a1de422e2162e40f8949e4b99291" exitCode=0
Jul 29 01:47:56.021170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:56.020822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh" event={"ID":"6cbc34cd-60de-4f93-8f19-1399f6ba047c","Type":"ContainerDied","Data":"7a1b2f30adc25b627d9e2f8da4783e43f631a1de422e2162e40f8949e4b99291"}
Jul 29 01:47:56.021170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:56.020882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh" event={"ID":"6cbc34cd-60de-4f93-8f19-1399f6ba047c","Type":"ContainerStarted","Data":"ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f"}
Jul 29 01:47:57.146939 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:57.146917    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:57.213257 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:57.213231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca\") pod \"6cbc34cd-60de-4f93-8f19-1399f6ba047c\" (UID: \"6cbc34cd-60de-4f93-8f19-1399f6ba047c\") "
Jul 29 01:47:57.213554 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:57.213531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cbc34cd-60de-4f93-8f19-1399f6ba047c" (UID: "6cbc34cd-60de-4f93-8f19-1399f6ba047c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:47:57.314543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:57.314521    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbc34cd-60de-4f93-8f19-1399f6ba047c-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:47:58.031639 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:58.031605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh" event={"ID":"6cbc34cd-60de-4f93-8f19-1399f6ba047c","Type":"ContainerDied","Data":"ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f"}
Jul 29 01:47:58.031639 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:58.031628    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"
Jul 29 01:47:58.031880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:47:58.031661    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1556566c662d7b9397070328b83be0277d328217074a3b5a5e4c28c25aa56f"
Jul 29 01:48:05.503282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:05.503253    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"]
Jul 29 01:48:05.507400 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:05.507376    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-689dd84c-jgwkh"]
Jul 29 01:48:07.465979 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:07.465944    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cbc34cd-60de-4f93-8f19-1399f6ba047c" path="/var/lib/kubelet/pods/6cbc34cd-60de-4f93-8f19-1399f6ba047c/volumes"
Jul 29 01:48:08.652829 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:08.652799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:48:35.676639 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.676604    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.676932    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cbc34cd-60de-4f93-8f19-1399f6ba047c" containerName="revoke-keys"
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.676942    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cbc34cd-60de-4f93-8f19-1399f6ba047c" containerName="revoke-keys"
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.676956    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.676961    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.677015    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cbc34cd-60de-4f93-8f19-1399f6ba047c" containerName="revoke-keys"
Jul 29 01:48:35.677085 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.677024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcaa63f3-c5d4-43c3-ba4e-ca6f1405eba1" containerName="istio-proxy"
Jul 29 01:48:35.680111 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.680094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.683672 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.683651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12321b47-dockercfg-6l5n2\""
Jul 29 01:48:35.683793 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.683777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12321b47\""
Jul 29 01:48:35.692941 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.692921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:48:35.709587 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.709492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.709698 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.709612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926vv\" (UniqueName: \"kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.810480 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.810452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926vv\" (UniqueName: \"kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.810639 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.810496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.811188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.811169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.818384 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.818368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926vv\" (UniqueName: \"kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv\") pod \"payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:35.990173 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:35.990112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:36.112519 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:36.112490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:48:36.114870 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:48:36.114844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cde587_cfe1_4565_91c5_422f4a81c8c4.slice/crio-d5f785a9a586b726785ccf591468dd05ad91ac5597d7a36324d8bfce30ccb3cd WatchSource:0}: Error finding container d5f785a9a586b726785ccf591468dd05ad91ac5597d7a36324d8bfce30ccb3cd: Status 404 returned error can't find the container with id d5f785a9a586b726785ccf591468dd05ad91ac5597d7a36324d8bfce30ccb3cd
Jul 29 01:48:36.117189 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:36.117170    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 01:48:36.199725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:36.199695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" event={"ID":"c9cde587-cfe1-4565-91c5-422f4a81c8c4","Type":"ContainerStarted","Data":"d5f785a9a586b726785ccf591468dd05ad91ac5597d7a36324d8bfce30ccb3cd"}
Jul 29 01:48:38.211109 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:38.211069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" event={"ID":"c9cde587-cfe1-4565-91c5-422f4a81c8c4","Type":"ContainerStarted","Data":"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"}
Jul 29 01:48:38.211533 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:38.211135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:48:38.226773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:38.226728    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" podStartSLOduration=1.7389512200000001 podStartE2EDuration="3.226714424s" totalImagesPullingTime="1.487763204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:38.22621168 +0000 UTC m=+2305.379657091" watchObservedRunningTime="2026-07-29 01:48:38.226714424 +0000 UTC m=+2305.380159799"
Jul 29 01:48:39.749246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:39.749211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:48:49.217031 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:48:49.217002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:49:05.756556 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.756480    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:49:05.760739 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.760711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.763449 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.763427    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12321b47\"/\"kube-root-ca.crt\""
Jul 29 01:49:05.764554 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.764535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12321b47\"/\"e2e-lifecycle-model-12321b47-kserve-self-signed-certs\""
Jul 29 01:49:05.764641 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.764554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12321b47\"/\"openshift-service-ca.crt\""
Jul 29 01:49:05.764641 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.764562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12321b47\"/\"default-dockercfg-ng8vh\""
Jul 29 01:49:05.769571 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.769549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:49:05.866769 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.866736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.866769 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.866767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.866999 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.866800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qjp\" (UniqueName: \"kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.866999 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.866825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.866999 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.866919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.867177 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.867008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.967788 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.967901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.967901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.967901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qjp\" (UniqueName: \"kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.967901 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.968068 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.967906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.968246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.968224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.968291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.968260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.968379 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.968364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.970109 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.970079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.970229 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.970212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:05.976429 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:05.976407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qjp\" (UniqueName: \"kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp\") pod \"e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") " pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:06.071614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:06.071589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:06.197837 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:06.197809    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:49:06.200063 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:49:06.200011    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b3e571a_5431_4628_aa41_d07a954347f1.slice/crio-ae49bb5d4307c43a9b9b27ca7910d20fd40c8ec83d25d94b07d5e5db9f33f307 WatchSource:0}: Error finding container ae49bb5d4307c43a9b9b27ca7910d20fd40c8ec83d25d94b07d5e5db9f33f307: Status 404 returned error can't find the container with id ae49bb5d4307c43a9b9b27ca7910d20fd40c8ec83d25d94b07d5e5db9f33f307
Jul 29 01:49:06.336612 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:06.336524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" event={"ID":"9b3e571a-5431-4628-aa41-d07a954347f1","Type":"ContainerStarted","Data":"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"}
Jul 29 01:49:06.336612 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:06.336573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" event={"ID":"9b3e571a-5431-4628-aa41-d07a954347f1","Type":"ContainerStarted","Data":"ae49bb5d4307c43a9b9b27ca7910d20fd40c8ec83d25d94b07d5e5db9f33f307"}
Jul 29 01:49:06.353992 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:06.353949    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" podStartSLOduration=1.353934129 podStartE2EDuration="1.353934129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:49:06.352451406 +0000 UTC m=+2333.505896782" watchObservedRunningTime="2026-07-29 01:49:06.353934129 +0000 UTC m=+2333.507379505"
Jul 29 01:49:08.947897 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:08.947863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:49:13.498147 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:13.498119    2577 scope.go:117] "RemoveContainer" containerID="e23530e8729a28e086bd994db29827277126b2721867494a5ded483126cd8dc7"
Jul 29 01:49:16.071812 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:16.071780    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:16.072275 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:16.071827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:16.083762 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:16.083741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:16.392067 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:16.391965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:49:23.465860 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:23.465821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:49:38.102526 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.102491    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:49:38.103145 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.102698    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" podUID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" containerName="payload-pre-processing" containerID="cri-o://aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87" gracePeriod=30
Jul 29 01:49:38.347543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.347518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:49:38.421474 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.421396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-926vv\" (UniqueName: \"kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv\") pod \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") "
Jul 29 01:49:38.421603 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.421480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume\") pod \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\" (UID: \"c9cde587-cfe1-4565-91c5-422f4a81c8c4\") "
Jul 29 01:49:38.421834 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.421807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9cde587-cfe1-4565-91c5-422f4a81c8c4" (UID: "c9cde587-cfe1-4565-91c5-422f4a81c8c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:49:38.423327 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.423307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv" (OuterVolumeSpecName: "kube-api-access-926vv") pod "c9cde587-cfe1-4565-91c5-422f4a81c8c4" (UID: "c9cde587-cfe1-4565-91c5-422f4a81c8c4"). InnerVolumeSpecName "kube-api-access-926vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:49:38.476711 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.476681    2577 generic.go:356] "Generic (PLEG): container finished" podID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" containerID="aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87" exitCode=0
Jul 29 01:49:38.476821 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.476724    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"
Jul 29 01:49:38.476821 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.476769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" event={"ID":"c9cde587-cfe1-4565-91c5-422f4a81c8c4","Type":"ContainerDied","Data":"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"}
Jul 29 01:49:38.476931 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.476819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp" event={"ID":"c9cde587-cfe1-4565-91c5-422f4a81c8c4","Type":"ContainerDied","Data":"d5f785a9a586b726785ccf591468dd05ad91ac5597d7a36324d8bfce30ccb3cd"}
Jul 29 01:49:38.476931 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.476849    2577 scope.go:117] "RemoveContainer" containerID="aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"
Jul 29 01:49:38.485584 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.485567    2577 scope.go:117] "RemoveContainer" containerID="aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"
Jul 29 01:49:38.485822 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:49:38.485804    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87\": container with ID starting with aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87 not found: ID does not exist" containerID="aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"
Jul 29 01:49:38.485885 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.485831    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87"} err="failed to get container status \"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87\": rpc error: code = NotFound desc = could not find container \"aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87\": container with ID starting with aa987e63aa76565c2a1ec09e3f57e80320d69575e7c3d0567a5d34170b6afc87 not found: ID does not exist"
Jul 29 01:49:38.497262 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.497242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:49:38.499485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.499466    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12321b47-6465ff867b-kqtkp"]
Jul 29 01:49:38.522358 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.522334    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9cde587-cfe1-4565-91c5-422f4a81c8c4-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:49:38.522358 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:38.522356    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926vv\" (UniqueName: \"kubernetes.io/projected/c9cde587-cfe1-4565-91c5-422f4a81c8c4-kube-api-access-926vv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:49:39.061254 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:39.061222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:49:39.466131 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:39.466102    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" path="/var/lib/kubelet/pods/c9cde587-cfe1-4565-91c5-422f4a81c8c4/volumes"
Jul 29 01:49:46.585741 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:46.585710    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:49:47.755097 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:49:47.755068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:50:01.604196 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.604153    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" podUID="9b3e571a-5431-4628-aa41-d07a954347f1" containerName="main" containerID="cri-o://9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a" gracePeriod=45
Jul 29 01:50:01.849981 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.849956    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:50:01.998652 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.998652 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998630    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49qjp\" (UniqueName: \"kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.998652 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998683    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir\") pod \"9b3e571a-5431-4628-aa41-d07a954347f1\" (UID: \"9b3e571a-5431-4628-aa41-d07a954347f1\") "
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.998930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.999025    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.999071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home" (OuterVolumeSpecName: "home") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:50:01.999250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:01.999137    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:50:02.000796 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.000765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:50:02.000796 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.000785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm" (OuterVolumeSpecName: "dshm") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:50:02.000932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.000822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp" (OuterVolumeSpecName: "kube-api-access-49qjp") pod "9b3e571a-5431-4628-aa41-d07a954347f1" (UID: "9b3e571a-5431-4628-aa41-d07a954347f1"). InnerVolumeSpecName "kube-api-access-49qjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:50:02.099390 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.099359    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:02.099390 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.099384    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:02.099390 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.099394    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3e571a-5431-4628-aa41-d07a954347f1-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:02.099578 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.099402    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qjp\" (UniqueName: \"kubernetes.io/projected/9b3e571a-5431-4628-aa41-d07a954347f1-kube-api-access-49qjp\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:02.099578 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.099410    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3e571a-5431-4628-aa41-d07a954347f1-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:50:02.582817 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.582781    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b3e571a-5431-4628-aa41-d07a954347f1" containerID="9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a" exitCode=0
Jul 29 01:50:02.583020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.582830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"
Jul 29 01:50:02.583020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.582864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" event={"ID":"9b3e571a-5431-4628-aa41-d07a954347f1","Type":"ContainerDied","Data":"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"}
Jul 29 01:50:02.583020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.582910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28" event={"ID":"9b3e571a-5431-4628-aa41-d07a954347f1","Type":"ContainerDied","Data":"ae49bb5d4307c43a9b9b27ca7910d20fd40c8ec83d25d94b07d5e5db9f33f307"}
Jul 29 01:50:02.583020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.582928    2577 scope.go:117] "RemoveContainer" containerID="9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"
Jul 29 01:50:02.591702 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.591686    2577 scope.go:117] "RemoveContainer" containerID="9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"
Jul 29 01:50:02.591948 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:50:02.591927    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a\": container with ID starting with 9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a not found: ID does not exist" containerID="9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"
Jul 29 01:50:02.592010 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.591960    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a"} err="failed to get container status \"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a\": rpc error: code = NotFound desc = could not find container \"9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a\": container with ID starting with 9ca27e0d443ec8f63eac209105ceb4aaf3051c27e6c010546bc73e4ee45e4b9a not found: ID does not exist"
Jul 29 01:50:02.608475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.608456    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:50:02.610850 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:02.610832    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12321b47/e2e-lifecycle-model-12321b47-kserve-7445fdcf95-q9g28"]
Jul 29 01:50:03.466390 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:03.466358    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3e571a-5431-4628-aa41-d07a954347f1" path="/var/lib/kubelet/pods/9b3e571a-5431-4628-aa41-d07a954347f1/volumes"
Jul 29 01:50:19.265849 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:19.265804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:50:31.980460 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:50:31.980426    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:51:13.530655 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:13.530618    2577 scope.go:117] "RemoveContainer" containerID="b98b94997207eb5ad1c0f383b3f294cefcdf02e8720083a3d71cc571d42fc96e"
Jul 29 01:51:14.488030 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.487994    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:51:14.488357 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488345    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3e571a-5431-4628-aa41-d07a954347f1" containerName="main"
Jul 29 01:51:14.488404 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488359    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3e571a-5431-4628-aa41-d07a954347f1" containerName="main"
Jul 29 01:51:14.488404 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" containerName="payload-pre-processing"
Jul 29 01:51:14.488404 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488378    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" containerName="payload-pre-processing"
Jul 29 01:51:14.488499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488434    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cde587-cfe1-4565-91c5-422f4a81c8c4" containerName="payload-pre-processing"
Jul 29 01:51:14.488499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.488443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3e571a-5431-4628-aa41-d07a954347f1" containerName="main"
Jul 29 01:51:14.492554 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.492537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.495314 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.495294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-560096\""
Jul 29 01:51:14.495466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.495296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-560096-dockercfg-7hzw6\""
Jul 29 01:51:14.505018 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.504994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:51:14.509259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.509238    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:51:14.512485 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.512467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.526551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.526529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:51:14.573163 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.573131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvt4\" (UniqueName: \"kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.573441 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.573176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.673773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.673743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.673886 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.673793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflwx\" (UniqueName: \"kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.673886 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.673853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvt4\" (UniqueName: \"kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.674010 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.673914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.674563 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.674545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.682538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.682518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvt4\" (UniqueName: \"kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4\") pod \"payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.774524 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.774464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.774524 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.774511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflwx\" (UniqueName: \"kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.775158 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.775142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.782414 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.782397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflwx\" (UniqueName: \"kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx\") pod \"payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.802205 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.802183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:14.825365 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.825343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:14.939973 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.939947    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:51:14.942489 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:51:14.942459    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f12f3d_d6c9_4d9a_aa4f_b6612b2752e9.slice/crio-7d35fba8edea4eefee2bfd982a5e942ce0bfafa014089dfb0012d4ffe934069a WatchSource:0}: Error finding container 7d35fba8edea4eefee2bfd982a5e942ce0bfafa014089dfb0012d4ffe934069a: Status 404 returned error can't find the container with id 7d35fba8edea4eefee2bfd982a5e942ce0bfafa014089dfb0012d4ffe934069a
Jul 29 01:51:14.964582 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:14.964557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:51:14.979028 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:51:14.979002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfa49a0_3954_4265_823d_2b35840406d9.slice/crio-3e5a1b204fe6ff257515d3d6705bccd64f6dd9edab5f874b98d1de204ed965b8 WatchSource:0}: Error finding container 3e5a1b204fe6ff257515d3d6705bccd64f6dd9edab5f874b98d1de204ed965b8: Status 404 returned error can't find the container with id 3e5a1b204fe6ff257515d3d6705bccd64f6dd9edab5f874b98d1de204ed965b8
Jul 29 01:51:15.930699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.930666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" event={"ID":"dcfa49a0-3954-4265-823d-2b35840406d9","Type":"ContainerStarted","Data":"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"}
Jul 29 01:51:15.931164 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.930716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" event={"ID":"dcfa49a0-3954-4265-823d-2b35840406d9","Type":"ContainerStarted","Data":"3e5a1b204fe6ff257515d3d6705bccd64f6dd9edab5f874b98d1de204ed965b8"}
Jul 29 01:51:15.931164 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.930801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:15.932162 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.932140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" event={"ID":"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9","Type":"ContainerStarted","Data":"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"}
Jul 29 01:51:15.932278 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.932175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" event={"ID":"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9","Type":"ContainerStarted","Data":"7d35fba8edea4eefee2bfd982a5e942ce0bfafa014089dfb0012d4ffe934069a"}
Jul 29 01:51:15.932278 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.932262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:15.949665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.949609    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" podStartSLOduration=1.949592584 podStartE2EDuration="1.949592584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:51:15.947777223 +0000 UTC m=+2463.101222603" watchObservedRunningTime="2026-07-29 01:51:15.949592584 +0000 UTC m=+2463.103037962"
Jul 29 01:51:15.964507 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:15.964464    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" podStartSLOduration=1.964448397 podStartE2EDuration="1.964448397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:51:15.963129903 +0000 UTC m=+2463.116575280" watchObservedRunningTime="2026-07-29 01:51:15.964448397 +0000 UTC m=+2463.117893777"
Jul 29 01:51:20.254600 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:20.254563    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:51:26.938550 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:26.938518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:51:26.938915 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:26.938605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:51:43.702930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.702901    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:51:43.706353 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.706337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.719332 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.719311    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ee5b9-dockercfg-xxntm\""
Jul 29 01:51:43.719551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.719535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ee5b9\""
Jul 29 01:51:43.732270 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.732242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:51:43.789215 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.789185    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:51:43.792506 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.792491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:43.801608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.801570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.801608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.801604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7x89\" (UniqueName: \"kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.808892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.808874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:51:43.902149 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.902116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.902149 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.902148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7x89\" (UniqueName: \"kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.902367 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.902200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:43.902367 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.902225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhqv\" (UniqueName: \"kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:43.902910 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.902874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:43.914299 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:43.914274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7x89\" (UniqueName: \"kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89\") pod \"payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:44.003142 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.003076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:44.003142 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.003107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhqv\" (UniqueName: \"kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:44.003763 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.003741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:44.017005 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.016985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:44.020251 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.020224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhqv\" (UniqueName: \"kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv\") pod \"payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:44.102347 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.102313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:44.151815 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.151782    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:51:44.155171 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:51:44.155134    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea381725_0d22_4541_a2d5_ca72ae071101.slice/crio-ee913e136acae69218f9c84e8b783d252a3690b8a08b97a8d4fad32bf26c56fb WatchSource:0}: Error finding container ee913e136acae69218f9c84e8b783d252a3690b8a08b97a8d4fad32bf26c56fb: Status 404 returned error can't find the container with id ee913e136acae69218f9c84e8b783d252a3690b8a08b97a8d4fad32bf26c56fb
Jul 29 01:51:44.246823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:44.246800    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:51:44.249233 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:51:44.249209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80705e7a_b457_48eb_a992_d57389ec5c72.slice/crio-b10ffbb52db36ed47280b8180ecbd2132f085f918d55c63dfc122334d5a12ef5 WatchSource:0}: Error finding container b10ffbb52db36ed47280b8180ecbd2132f085f918d55c63dfc122334d5a12ef5: Status 404 returned error can't find the container with id b10ffbb52db36ed47280b8180ecbd2132f085f918d55c63dfc122334d5a12ef5
Jul 29 01:51:45.060261 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.060229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" event={"ID":"80705e7a-b457-48eb-a992-d57389ec5c72","Type":"ContainerStarted","Data":"3acf4368373533249d7f6e89d1341ce11c1e2ee511394ef20b8ad31050ba35e1"}
Jul 29 01:51:45.060743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.060286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" event={"ID":"80705e7a-b457-48eb-a992-d57389ec5c72","Type":"ContainerStarted","Data":"b10ffbb52db36ed47280b8180ecbd2132f085f918d55c63dfc122334d5a12ef5"}
Jul 29 01:51:45.060743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.060377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:51:45.061764 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.061736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" event={"ID":"ea381725-0d22-4541-a2d5-ca72ae071101","Type":"ContainerStarted","Data":"c8721b804e0c0ada4114fdd150037a1b45d1801c66b0a8284486998e99350a92"}
Jul 29 01:51:45.061892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.061784    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:45.061892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.061803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" event={"ID":"ea381725-0d22-4541-a2d5-ca72ae071101","Type":"ContainerStarted","Data":"ee913e136acae69218f9c84e8b783d252a3690b8a08b97a8d4fad32bf26c56fb"}
Jul 29 01:51:45.078259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.078216    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" podStartSLOduration=2.078201903 podStartE2EDuration="2.078201903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:51:45.076610402 +0000 UTC m=+2492.230055790" watchObservedRunningTime="2026-07-29 01:51:45.078201903 +0000 UTC m=+2492.231647257"
Jul 29 01:51:45.094288 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:45.094249    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" podStartSLOduration=2.094237062 podStartE2EDuration="2.094237062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:51:45.092436357 +0000 UTC m=+2492.245881736" watchObservedRunningTime="2026-07-29 01:51:45.094237062 +0000 UTC m=+2492.247682438"
Jul 29 01:51:50.358783 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:50.358748    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:51:56.068062 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:56.068022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:51:56.068428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:51:56.068106    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:52:26.259734 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.259660    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:52:26.262962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.262943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.266800 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.266773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-560096\"/\"openshift-service-ca.crt\""
Jul 29 01:52:26.266925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.266778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-560096\"/\"default-dockercfg-nmvjh\""
Jul 29 01:52:26.266925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.266778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-560096\"/\"kube-root-ca.crt\""
Jul 29 01:52:26.266925 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.266834    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-560096\"/\"test-model-560096-kserve-self-signed-certs\""
Jul 29 01:52:26.275310 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.275288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:52:26.332285 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvwv\" (UniqueName: \"kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.332419 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.332419 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.332419 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.332576 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.332576 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.332486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.433662 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.433823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.433823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvwv\" (UniqueName: \"kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.433948 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.433948 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.434080 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.433947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.434132 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.434120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.434258 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.434238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.434316 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.434295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.435979 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.435959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.436250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.436233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.441629 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.441610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvwv\" (UniqueName: \"kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv\") pod \"test-model-560096-kserve-84c8d5fb5-frwkm\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") " pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.572384 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.572358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:26.699655 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:26.699631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:52:26.702110 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:52:26.702077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode76457bc_cd83_40a6_9189_53ab9218eb5f.slice/crio-61d97ed3e50f9e25f6e1ce03ec1823f9900a35993def863fea927ac5336a27b7 WatchSource:0}: Error finding container 61d97ed3e50f9e25f6e1ce03ec1823f9900a35993def863fea927ac5336a27b7: Status 404 returned error can't find the container with id 61d97ed3e50f9e25f6e1ce03ec1823f9900a35993def863fea927ac5336a27b7
Jul 29 01:52:27.250743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:27.250704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" event={"ID":"e76457bc-cd83-40a6-9189-53ab9218eb5f","Type":"ContainerStarted","Data":"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"}
Jul 29 01:52:27.250743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:27.250745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" event={"ID":"e76457bc-cd83-40a6-9189-53ab9218eb5f","Type":"ContainerStarted","Data":"61d97ed3e50f9e25f6e1ce03ec1823f9900a35993def863fea927ac5336a27b7"}
Jul 29 01:52:27.269505 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:27.269458    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" podStartSLOduration=1.26944304 podStartE2EDuration="1.26944304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:52:27.268885163 +0000 UTC m=+2534.422330538" watchObservedRunningTime="2026-07-29 01:52:27.26944304 +0000 UTC m=+2534.422888416"
Jul 29 01:52:29.355023 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:29.354994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:52:36.573001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:36.572962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:36.573482 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:36.573020    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:36.586464 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:36.586437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:37.313857 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:37.313828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:52:41.853363 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:41.853325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:52:54.464785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.464751    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:52:54.468191 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.468170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.474400 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.474380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ee5b9\"/\"kube-root-ca.crt\""
Jul 29 01:52:54.475826 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.475806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ee5b9\"/\"default-dockercfg-k9rxp\""
Jul 29 01:52:54.475928 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.475838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ee5b9\"/\"openshift-service-ca.crt\""
Jul 29 01:52:54.475928 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.475806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ee5b9\"/\"test-model-5ee5b9-kserve-self-signed-certs\""
Jul 29 01:52:54.482524 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.482502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:52:54.546773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.546937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.546937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866kw\" (UniqueName: \"kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.546937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.546937 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.547139 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.546954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647509 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647679 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647679 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866kw\" (UniqueName: \"kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647679 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647679 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.647679 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.648003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.648003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.647992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.648155 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.648024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.649696 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.649679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.650183 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.650166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.655795 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.655767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-866kw\" (UniqueName: \"kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw\") pod \"test-model-5ee5b9-kserve-668b887d99-5vrmx\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") " pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.777468 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.777391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:52:54.904174 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:54.904152    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:52:54.906550 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:52:54.906520    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab270eee_39f5_4a9a_9472_a67517ba7888.slice/crio-5e27d40b4bd1258eb928b6c50f6fef3e617d3f19e0904be8b55f4223a82a5908 WatchSource:0}: Error finding container 5e27d40b4bd1258eb928b6c50f6fef3e617d3f19e0904be8b55f4223a82a5908: Status 404 returned error can't find the container with id 5e27d40b4bd1258eb928b6c50f6fef3e617d3f19e0904be8b55f4223a82a5908
Jul 29 01:52:55.390910 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:55.390876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" event={"ID":"ab270eee-39f5-4a9a-9472-a67517ba7888","Type":"ContainerStarted","Data":"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"}
Jul 29 01:52:55.391095 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:55.390919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" event={"ID":"ab270eee-39f5-4a9a-9472-a67517ba7888","Type":"ContainerStarted","Data":"5e27d40b4bd1258eb928b6c50f6fef3e617d3f19e0904be8b55f4223a82a5908"}
Jul 29 01:52:55.410187 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:52:55.410145    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" podStartSLOduration=1.410131878 podStartE2EDuration="1.410131878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:52:55.409428738 +0000 UTC m=+2562.562874149" watchObservedRunningTime="2026-07-29 01:52:55.410131878 +0000 UTC m=+2562.563577299"
Jul 29 01:53:00.157773 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:00.157737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:53:04.778619 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:04.778584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:04.778619 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:04.778628    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:04.790785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:04.790762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:05.447239 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:05.447203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:32.333873 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:32.333837    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:53:33.246725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:33.246691    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:53:37.253730 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:37.253695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:53:43.873606 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:43.873570    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:53:43.874120 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:43.873766    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" podUID="dcfa49a0-3954-4265-823d-2b35840406d9" containerName="payload-processing" containerID="cri-o://5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4" gracePeriod=30
Jul 29 01:53:43.879505 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:43.879475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:53:43.880379 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:43.879678    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" podUID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" containerName="payload-pre-processing" containerID="cri-o://08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2" gracePeriod=30
Jul 29 01:53:44.136311 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.136286    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:53:44.152094 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.152073    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:53:44.256125 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume\") pod \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") "
Jul 29 01:53:44.256271 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnvt4\" (UniqueName: \"kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4\") pod \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\" (UID: \"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9\") "
Jul 29 01:53:44.256271 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume\") pod \"dcfa49a0-3954-4265-823d-2b35840406d9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") "
Jul 29 01:53:44.256271 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nflwx\" (UniqueName: \"kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx\") pod \"dcfa49a0-3954-4265-823d-2b35840406d9\" (UID: \"dcfa49a0-3954-4265-823d-2b35840406d9\") "
Jul 29 01:53:44.256549 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256523    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" (UID: "d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:53:44.256609 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.256584    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcfa49a0-3954-4265-823d-2b35840406d9" (UID: "dcfa49a0-3954-4265-823d-2b35840406d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:53:44.258312 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.258284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx" (OuterVolumeSpecName: "kube-api-access-nflwx") pod "dcfa49a0-3954-4265-823d-2b35840406d9" (UID: "dcfa49a0-3954-4265-823d-2b35840406d9"). InnerVolumeSpecName "kube-api-access-nflwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:53:44.258394 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.258327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4" (OuterVolumeSpecName: "kube-api-access-hnvt4") pod "d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" (UID: "d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9"). InnerVolumeSpecName "kube-api-access-hnvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:53:44.357598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.357571    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflwx\" (UniqueName: \"kubernetes.io/projected/dcfa49a0-3954-4265-823d-2b35840406d9-kube-api-access-nflwx\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:44.357598 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.357595    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:44.357751 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.357605    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvt4\" (UniqueName: \"kubernetes.io/projected/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9-kube-api-access-hnvt4\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:44.357751 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.357614    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfa49a0-3954-4265-823d-2b35840406d9-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:44.613267 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.613235    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" containerID="08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2" exitCode=0
Jul 29 01:53:44.613428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.613281    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"
Jul 29 01:53:44.613428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.613321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" event={"ID":"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9","Type":"ContainerDied","Data":"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"}
Jul 29 01:53:44.613428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.613373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st" event={"ID":"d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9","Type":"ContainerDied","Data":"7d35fba8edea4eefee2bfd982a5e942ce0bfafa014089dfb0012d4ffe934069a"}
Jul 29 01:53:44.613428 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.613399    2577 scope.go:117] "RemoveContainer" containerID="08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"
Jul 29 01:53:44.614673 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.614652    2577 generic.go:356] "Generic (PLEG): container finished" podID="dcfa49a0-3954-4265-823d-2b35840406d9" containerID="5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4" exitCode=0
Jul 29 01:53:44.614792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.614685    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"
Jul 29 01:53:44.614792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.614696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" event={"ID":"dcfa49a0-3954-4265-823d-2b35840406d9","Type":"ContainerDied","Data":"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"}
Jul 29 01:53:44.614792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.614729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9" event={"ID":"dcfa49a0-3954-4265-823d-2b35840406d9","Type":"ContainerDied","Data":"3e5a1b204fe6ff257515d3d6705bccd64f6dd9edab5f874b98d1de204ed965b8"}
Jul 29 01:53:44.622971 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.622960    2577 scope.go:117] "RemoveContainer" containerID="08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"
Jul 29 01:53:44.623299 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:53:44.623254    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2\": container with ID starting with 08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2 not found: ID does not exist" containerID="08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"
Jul 29 01:53:44.623421 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.623292    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2"} err="failed to get container status \"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2\": rpc error: code = NotFound desc = could not find container \"08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2\": container with ID starting with 08571c02ee65226040eb63f6c6d10273f5ce6ce81b922be2509bb69d65feede2 not found: ID does not exist"
Jul 29 01:53:44.623421 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.623316    2577 scope.go:117] "RemoveContainer" containerID="5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"
Jul 29 01:53:44.630689 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.630673    2577 scope.go:117] "RemoveContainer" containerID="5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"
Jul 29 01:53:44.630942 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:53:44.630923    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4\": container with ID starting with 5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4 not found: ID does not exist" containerID="5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"
Jul 29 01:53:44.631029 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.630947    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4"} err="failed to get container status \"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4\": rpc error: code = NotFound desc = could not find container \"5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4\": container with ID starting with 5ae40792ed1f3a8f8ffacf1d1f335b873f2e37727bbc0d2f0980679ddaf51be4 not found: ID does not exist"
Jul 29 01:53:44.639389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.639364    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:53:44.643725 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.643693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-560096-7f844f6db6-vnth9"]
Jul 29 01:53:44.654348 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.654322    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:53:44.657774 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:44.657754    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-560096-9cf8cb96-f76st"]
Jul 29 01:53:45.466471 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:45.466442    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" path="/var/lib/kubelet/pods/d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9/volumes"
Jul 29 01:53:45.466845 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:45.466786    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcfa49a0-3954-4265-823d-2b35840406d9" path="/var/lib/kubelet/pods/dcfa49a0-3954-4265-823d-2b35840406d9/volumes"
Jul 29 01:53:47.352320 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.352280    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" podUID="e76457bc-cd83-40a6-9189-53ab9218eb5f" containerName="main" containerID="cri-o://732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51" gracePeriod=45
Jul 29 01:53:47.598513 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.598484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:53:47.631850 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.631783    2577 generic.go:356] "Generic (PLEG): container finished" podID="e76457bc-cd83-40a6-9189-53ab9218eb5f" containerID="732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51" exitCode=0
Jul 29 01:53:47.631850 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.631827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"
Jul 29 01:53:47.632033 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.631820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" event={"ID":"e76457bc-cd83-40a6-9189-53ab9218eb5f","Type":"ContainerDied","Data":"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"}
Jul 29 01:53:47.632033 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.631959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm" event={"ID":"e76457bc-cd83-40a6-9189-53ab9218eb5f","Type":"ContainerDied","Data":"61d97ed3e50f9e25f6e1ce03ec1823f9900a35993def863fea927ac5336a27b7"}
Jul 29 01:53:47.632033 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.631987    2577 scope.go:117] "RemoveContainer" containerID="732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"
Jul 29 01:53:47.639858 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.639842    2577 scope.go:117] "RemoveContainer" containerID="732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"
Jul 29 01:53:47.640150 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:53:47.640120    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51\": container with ID starting with 732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51 not found: ID does not exist" containerID="732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"
Jul 29 01:53:47.640250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.640157    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51"} err="failed to get container status \"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51\": rpc error: code = NotFound desc = could not find container \"732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51\": container with ID starting with 732bd61e4472be19d2540ee611d45078804261348e34cce5be335b64ae23be51 not found: ID does not exist"
Jul 29 01:53:47.786741 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.786895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.786895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.786895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.786895 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bvwv\" (UniqueName: \"kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.786941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs\") pod \"e76457bc-cd83-40a6-9189-53ab9218eb5f\" (UID: \"e76457bc-cd83-40a6-9189-53ab9218eb5f\") "
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home" (OuterVolumeSpecName: "home") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787278    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.787297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787298    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.787584 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.787311    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.789003 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.788973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm" (OuterVolumeSpecName: "dshm") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:47.789120 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.789056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv" (OuterVolumeSpecName: "kube-api-access-4bvwv") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "kube-api-access-4bvwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:53:47.789170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.789135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e76457bc-cd83-40a6-9189-53ab9218eb5f" (UID: "e76457bc-cd83-40a6-9189-53ab9218eb5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:53:47.888648 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.888570    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bvwv\" (UniqueName: \"kubernetes.io/projected/e76457bc-cd83-40a6-9189-53ab9218eb5f-kube-api-access-4bvwv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.888648 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.888596    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76457bc-cd83-40a6-9189-53ab9218eb5f-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.888648 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.888605    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76457bc-cd83-40a6-9189-53ab9218eb5f-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:47.953881 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.953852    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:53:47.957718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:47.957697    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-560096/test-model-560096-kserve-84c8d5fb5-frwkm"]
Jul 29 01:53:48.265617 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.265541    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" podUID="ab270eee-39f5-4a9a-9472-a67517ba7888" containerName="main" containerID="cri-o://cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073" gracePeriod=45
Jul 29 01:53:48.524548 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.524493    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:48.643250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.643216    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab270eee-39f5-4a9a-9472-a67517ba7888" containerID="cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073" exitCode=0
Jul 29 01:53:48.643413 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.643263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"
Jul 29 01:53:48.643413 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.643303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" event={"ID":"ab270eee-39f5-4a9a-9472-a67517ba7888","Type":"ContainerDied","Data":"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"}
Jul 29 01:53:48.643413 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.643342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx" event={"ID":"ab270eee-39f5-4a9a-9472-a67517ba7888","Type":"ContainerDied","Data":"5e27d40b4bd1258eb928b6c50f6fef3e617d3f19e0904be8b55f4223a82a5908"}
Jul 29 01:53:48.643413 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.643359    2577 scope.go:117] "RemoveContainer" containerID="cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"
Jul 29 01:53:48.651313 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.651298    2577 scope.go:117] "RemoveContainer" containerID="cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"
Jul 29 01:53:48.651552 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:53:48.651537    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073\": container with ID starting with cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073 not found: ID does not exist" containerID="cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"
Jul 29 01:53:48.651592 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.651558    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073"} err="failed to get container status \"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073\": rpc error: code = NotFound desc = could not find container \"cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073\": container with ID starting with cf6331fa7902d6a04dd2c9f3fb2c20f1e1e28a2065542642ff008ff4a588c073 not found: ID does not exist"
Jul 29 01:53:48.695965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.695943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696063 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.695984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-866kw\" (UniqueName: \"kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696063 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696008    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696063 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache\") pod \"ab270eee-39f5-4a9a-9472-a67517ba7888\" (UID: \"ab270eee-39f5-4a9a-9472-a67517ba7888\") "
Jul 29 01:53:48.696311 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:48.696311 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home" (OuterVolumeSpecName: "home") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:48.696425 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696389    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.696425 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:48.696501 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.696417    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.698176 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.698152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm" (OuterVolumeSpecName: "dshm") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:53:48.698277 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.698223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 01:53:48.698334 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.698296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw" (OuterVolumeSpecName: "kube-api-access-866kw") pod "ab270eee-39f5-4a9a-9472-a67517ba7888" (UID: "ab270eee-39f5-4a9a-9472-a67517ba7888"). InnerVolumeSpecName "kube-api-access-866kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:53:48.797674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.797614    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866kw\" (UniqueName: \"kubernetes.io/projected/ab270eee-39f5-4a9a-9472-a67517ba7888-kube-api-access-866kw\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.797674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.797638    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab270eee-39f5-4a9a-9472-a67517ba7888-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.797674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.797651    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.797674 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.797661    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab270eee-39f5-4a9a-9472-a67517ba7888-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:48.964594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.964532    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:53:48.965984 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:48.965965    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5ee5b9/test-model-5ee5b9-kserve-668b887d99-5vrmx"]
Jul 29 01:53:49.466499 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:49.466468    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab270eee-39f5-4a9a-9472-a67517ba7888" path="/var/lib/kubelet/pods/ab270eee-39f5-4a9a-9472-a67517ba7888/volumes"
Jul 29 01:53:49.466841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:49.466827    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76457bc-cd83-40a6-9189-53ab9218eb5f" path="/var/lib/kubelet/pods/e76457bc-cd83-40a6-9189-53ab9218eb5f/volumes"
Jul 29 01:53:50.848259 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"]
Jul 29 01:53:50.848711 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848691    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76457bc-cd83-40a6-9189-53ab9218eb5f" containerName="main"
Jul 29 01:53:50.848799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76457bc-cd83-40a6-9189-53ab9218eb5f" containerName="main"
Jul 29 01:53:50.848799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848749    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" containerName="payload-pre-processing"
Jul 29 01:53:50.848799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848758    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" containerName="payload-pre-processing"
Jul 29 01:53:50.848799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848777    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab270eee-39f5-4a9a-9472-a67517ba7888" containerName="main"
Jul 29 01:53:50.848799 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848787    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab270eee-39f5-4a9a-9472-a67517ba7888" containerName="main"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848805    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcfa49a0-3954-4265-823d-2b35840406d9" containerName="payload-processing"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848815    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcfa49a0-3954-4265-823d-2b35840406d9" containerName="payload-processing"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848921    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcfa49a0-3954-4265-823d-2b35840406d9" containerName="payload-processing"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848935    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab270eee-39f5-4a9a-9472-a67517ba7888" containerName="main"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848945    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f12f3d-d6c9-4d9a-aa4f-b6612b2752e9" containerName="payload-pre-processing"
Jul 29 01:53:50.849035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.848956    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76457bc-cd83-40a6-9189-53ab9218eb5f" containerName="main"
Jul 29 01:53:50.851595 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.851575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:50.854287 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.854269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:53:50.854385 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.854314    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 01:53:50.859710 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:50.859688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"]
Jul 29 01:53:51.014010 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.013984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv\" (UID: \"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:51.114317 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.114247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv\" (UID: \"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:51.114824 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.114806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv\" (UID: \"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:51.163358 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.163336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:51.496000 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.495978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"]
Jul 29 01:53:51.497839 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:53:51.497796    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888ea3d3_2f5f_4bc2_ba61_498e83ab1eaa.slice/crio-ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052 WatchSource:0}: Error finding container ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052: Status 404 returned error can't find the container with id ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052
Jul 29 01:53:51.659608 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:51.659584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv" event={"ID":"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa","Type":"ContainerStarted","Data":"ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052"}
Jul 29 01:53:52.665794 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:52.665759    2577 generic.go:356] "Generic (PLEG): container finished" podID="888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" containerID="3770431e46a439a5b704cf126df31f3947b8e2a0911b1a3b4b146d93a3aaa5ec" exitCode=0
Jul 29 01:53:52.666170 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:52.665838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv" event={"ID":"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa","Type":"ContainerDied","Data":"3770431e46a439a5b704cf126df31f3947b8e2a0911b1a3b4b146d93a3aaa5ec"}
Jul 29 01:53:53.373376 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:53.373339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:53:53.792824 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:53.792797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:53.937389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:53.937313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca\") pod \"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa\" (UID: \"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa\") "
Jul 29 01:53:53.937677 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:53.937654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" (UID: "888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:53:54.038459 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:54.038434    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:53:54.675816 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:54.675782    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"
Jul 29 01:53:54.675985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:54.675828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv" event={"ID":"888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa","Type":"ContainerDied","Data":"ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052"}
Jul 29 01:53:54.675985 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:53:54.675877    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9acb0c4086341e696e453b18bda6de21f07bdec249217a0329121d2c2f8052"
Jul 29 01:54:01.543830 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.543799    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:54:01.544412 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.544032    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" podUID="80705e7a-b457-48eb-a992-d57389ec5c72" containerName="payload-processing" containerID="cri-o://3acf4368373533249d7f6e89d1341ce11c1e2ee511394ef20b8ad31050ba35e1" gracePeriod=30
Jul 29 01:54:01.552480 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.552452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:54:01.552743 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.552700    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" podUID="ea381725-0d22-4541-a2d5-ca72ae071101" containerName="payload-pre-processing" containerID="cri-o://c8721b804e0c0ada4114fdd150037a1b45d1801c66b0a8284486998e99350a92" gracePeriod=30
Jul 29 01:54:01.709821 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.709789    2577 generic.go:356] "Generic (PLEG): container finished" podID="ea381725-0d22-4541-a2d5-ca72ae071101" containerID="c8721b804e0c0ada4114fdd150037a1b45d1801c66b0a8284486998e99350a92" exitCode=0
Jul 29 01:54:01.709993 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.709926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" event={"ID":"ea381725-0d22-4541-a2d5-ca72ae071101","Type":"ContainerDied","Data":"c8721b804e0c0ada4114fdd150037a1b45d1801c66b0a8284486998e99350a92"}
Jul 29 01:54:01.711703 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.711612    2577 generic.go:356] "Generic (PLEG): container finished" podID="80705e7a-b457-48eb-a992-d57389ec5c72" containerID="3acf4368373533249d7f6e89d1341ce11c1e2ee511394ef20b8ad31050ba35e1" exitCode=0
Jul 29 01:54:01.711703 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.711651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" event={"ID":"80705e7a-b457-48eb-a992-d57389ec5c72","Type":"ContainerDied","Data":"3acf4368373533249d7f6e89d1341ce11c1e2ee511394ef20b8ad31050ba35e1"}
Jul 29 01:54:01.716607 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.716566    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"]
Jul 29 01:54:01.719468 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.719394    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ee5b9-l8hsv"]
Jul 29 01:54:01.830035 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.830011    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:54:01.832708 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.832690    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:54:01.896914 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.896883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7x89\" (UniqueName: \"kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89\") pod \"ea381725-0d22-4541-a2d5-ca72ae071101\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") "
Jul 29 01:54:01.897032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.896919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume\") pod \"ea381725-0d22-4541-a2d5-ca72ae071101\" (UID: \"ea381725-0d22-4541-a2d5-ca72ae071101\") "
Jul 29 01:54:01.897032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.896938    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume\") pod \"80705e7a-b457-48eb-a992-d57389ec5c72\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") "
Jul 29 01:54:01.897032 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.897007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhqv\" (UniqueName: \"kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv\") pod \"80705e7a-b457-48eb-a992-d57389ec5c72\" (UID: \"80705e7a-b457-48eb-a992-d57389ec5c72\") "
Jul 29 01:54:01.897338 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.897317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea381725-0d22-4541-a2d5-ca72ae071101" (UID: "ea381725-0d22-4541-a2d5-ca72ae071101"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:54:01.897416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.897383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80705e7a-b457-48eb-a992-d57389ec5c72" (UID: "80705e7a-b457-48eb-a992-d57389ec5c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:54:01.898998 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.898979    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89" (OuterVolumeSpecName: "kube-api-access-c7x89") pod "ea381725-0d22-4541-a2d5-ca72ae071101" (UID: "ea381725-0d22-4541-a2d5-ca72ae071101"). InnerVolumeSpecName "kube-api-access-c7x89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:54:01.899081 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.898994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv" (OuterVolumeSpecName: "kube-api-access-hfhqv") pod "80705e7a-b457-48eb-a992-d57389ec5c72" (UID: "80705e7a-b457-48eb-a992-d57389ec5c72"). InnerVolumeSpecName "kube-api-access-hfhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:54:01.998075 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.998025    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhqv\" (UniqueName: \"kubernetes.io/projected/80705e7a-b457-48eb-a992-d57389ec5c72-kube-api-access-hfhqv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:54:01.998075 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.998070    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7x89\" (UniqueName: \"kubernetes.io/projected/ea381725-0d22-4541-a2d5-ca72ae071101-kube-api-access-c7x89\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:54:01.998075 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.998080    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea381725-0d22-4541-a2d5-ca72ae071101-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:54:01.998280 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:01.998089    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80705e7a-b457-48eb-a992-d57389ec5c72-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:54:02.717184 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.717155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"
Jul 29 01:54:02.717635 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.717195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2" event={"ID":"ea381725-0d22-4541-a2d5-ca72ae071101","Type":"ContainerDied","Data":"ee913e136acae69218f9c84e8b783d252a3690b8a08b97a8d4fad32bf26c56fb"}
Jul 29 01:54:02.717635 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.717258    2577 scope.go:117] "RemoveContainer" containerID="c8721b804e0c0ada4114fdd150037a1b45d1801c66b0a8284486998e99350a92"
Jul 29 01:54:02.718597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.718492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4" event={"ID":"80705e7a-b457-48eb-a992-d57389ec5c72","Type":"ContainerDied","Data":"b10ffbb52db36ed47280b8180ecbd2132f085f918d55c63dfc122334d5a12ef5"}
Jul 29 01:54:02.718597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.718514    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"
Jul 29 01:54:02.726705 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.726685    2577 scope.go:117] "RemoveContainer" containerID="3acf4368373533249d7f6e89d1341ce11c1e2ee511394ef20b8ad31050ba35e1"
Jul 29 01:54:02.747299 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.747274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:54:02.751841 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.751819    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ee5b9-58f57b945d-94cc4"]
Jul 29 01:54:02.762785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.762765    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:54:02.768762 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:02.768742    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ee5b9-7d6c9c85bd-r9wp2"]
Jul 29 01:54:03.466648 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:03.466615    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80705e7a-b457-48eb-a992-d57389ec5c72" path="/var/lib/kubelet/pods/80705e7a-b457-48eb-a992-d57389ec5c72/volumes"
Jul 29 01:54:03.466930 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:03.466918    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" path="/var/lib/kubelet/pods/888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa/volumes"
Jul 29 01:54:03.467264 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:03.467251    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea381725-0d22-4541-a2d5-ca72ae071101" path="/var/lib/kubelet/pods/ea381725-0d22-4541-a2d5-ca72ae071101/volumes"
Jul 29 01:54:04.462515 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:04.462477    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:54:09.075397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075362    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075675    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea381725-0d22-4541-a2d5-ca72ae071101" containerName="payload-pre-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea381725-0d22-4541-a2d5-ca72ae071101" containerName="payload-pre-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" containerName="revoke-keys"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" containerName="revoke-keys"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075726    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80705e7a-b457-48eb-a992-d57389ec5c72" containerName="payload-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075732    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80705e7a-b457-48eb-a992-d57389ec5c72" containerName="payload-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075779    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea381725-0d22-4541-a2d5-ca72ae071101" containerName="payload-pre-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075787    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80705e7a-b457-48eb-a992-d57389ec5c72" containerName="payload-processing"
Jul 29 01:54:09.075832 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.075792    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="888ea3d3-2f5f-4bc2-ba61-498e83ab1eaa" containerName="revoke-keys"
Jul 29 01:54:09.080159 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.080142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.083389 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.083362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2f7b9a-openshift-default-dockercfg-v6zms\""
Jul 29 01:54:09.083920 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.083901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 01:54:09.099378 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.099358    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:54:09.145961 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.145937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.145973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.145996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146077 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhkd\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.146208 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.146199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247224 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247323 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247323 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247323 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247323 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhkd\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247538 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247750 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247750 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247853 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.247853 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.247837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.248113 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.248095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.250067 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.250018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.250256 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.250236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.256650 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.256626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhkd\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.256823 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.256798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token\") pod \"e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") " pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.389691 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.389623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:09.545530 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.545503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:54:09.546777 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:54:09.546754    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7081811_8a0e_40f8_8c60_ac4350ea0328.slice/crio-379f8219741b06fd0729c44a7bdbc24ca2577ab5b9cb5562a23bf6818e047ee4 WatchSource:0}: Error finding container 379f8219741b06fd0729c44a7bdbc24ca2577ab5b9cb5562a23bf6818e047ee4: Status 404 returned error can't find the container with id 379f8219741b06fd0729c44a7bdbc24ca2577ab5b9cb5562a23bf6818e047ee4
Jul 29 01:54:09.548896 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.548861    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:09.548984 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.548927    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:09.548984 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.548957    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:09.755162 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.755028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerStarted","Data":"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"}
Jul 29 01:54:09.755162 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.755123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerStarted","Data":"379f8219741b06fd0729c44a7bdbc24ca2577ab5b9cb5562a23bf6818e047ee4"}
Jul 29 01:54:09.779569 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:09.779523    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podStartSLOduration=0.779507892 podStartE2EDuration="779.507892ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:54:09.778129847 +0000 UTC m=+2636.931575225" watchObservedRunningTime="2026-07-29 01:54:09.779507892 +0000 UTC m=+2636.932953268"
Jul 29 01:54:10.390072 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:10.390023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:10.395403 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:10.395371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:10.759664 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:10.759581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:10.760702 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:10.760683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:12.777990 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.777958    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:54:12.781427 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.781410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.784317 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.784296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2f7b9a-dockercfg-4lhmf\""
Jul 29 01:54:12.784513 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.784499    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2f7b9a\""
Jul 29 01:54:12.796305 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.796282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:54:12.884378 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.884346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkmv\" (UniqueName: \"kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.884527 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.884494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.985276 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.985238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkmv\" (UniqueName: \"kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.985434 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.985323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.985975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.985952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:12.994612 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:12.994588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkmv\" (UniqueName: \"kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv\") pod \"payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:13.092641 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.092610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:13.221069 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.221022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:54:13.223721 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:54:13.223697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa9c737_f2f8_4d18_b400_ec4bf9202928.slice/crio-b202c3207e1a8a01a48c26a24efe2d2fd967e51d5c1692cd7ec1e1ae7c7cb3f0 WatchSource:0}: Error finding container b202c3207e1a8a01a48c26a24efe2d2fd967e51d5c1692cd7ec1e1ae7c7cb3f0: Status 404 returned error can't find the container with id b202c3207e1a8a01a48c26a24efe2d2fd967e51d5c1692cd7ec1e1ae7c7cb3f0
Jul 29 01:54:13.561958 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.561932    2577 scope.go:117] "RemoveContainer" containerID="7a1b2f30adc25b627d9e2f8da4783e43f631a1de422e2162e40f8949e4b99291"
Jul 29 01:54:13.775047 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.775012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" event={"ID":"8fa9c737-f2f8-4d18-b400-ec4bf9202928","Type":"ContainerStarted","Data":"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"}
Jul 29 01:54:13.775145 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.775086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" event={"ID":"8fa9c737-f2f8-4d18-b400-ec4bf9202928","Type":"ContainerStarted","Data":"b202c3207e1a8a01a48c26a24efe2d2fd967e51d5c1692cd7ec1e1ae7c7cb3f0"}
Jul 29 01:54:13.775145 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.775115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:13.794450 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:13.794408    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" podStartSLOduration=1.794394941 podStartE2EDuration="1.794394941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:54:13.792723884 +0000 UTC m=+2640.946169260" watchObservedRunningTime="2026-07-29 01:54:13.794394941 +0000 UTC m=+2640.947840316"
Jul 29 01:54:17.367551 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:17.367518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:54:24.781553 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:24.781523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:54:25.760955 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.760922    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 01:54:25.761188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.760976    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 01:54:25.833590 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.833559    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerID="04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab" exitCode=0
Jul 29 01:54:25.833926 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.833632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerDied","Data":"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"}
Jul 29 01:54:25.833988 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.833973    2577 scope.go:117] "RemoveContainer" containerID="04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"
Jul 29 01:54:25.835021 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.834991    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:25.835143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.835091    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:25.835143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:25.835137    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:54:26.842152 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:26.842114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerStarted","Data":"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"}
Jul 29 01:54:27.390811 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:27.390775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:27.391087 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:27.391028    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 01:54:27.391235 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:27.391110    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 01:54:28.390329 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:28.390292    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 01:54:28.390797 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:28.390351    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 01:54:29.390309 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:29.390277    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 01:54:29.390468 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:29.390357    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 01:54:30.390586 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:30.390549    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 29 01:54:30.390970 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:30.390622    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 29 01:54:31.394297 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:31.394262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:31.394730 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:31.394534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:31.395240 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:31.395223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:54:41.772382 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.772345    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:54:41.780557 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.780525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:41.787487 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.787466    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9132e3\""
Jul 29 01:54:41.795241 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.795210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:54:41.927013 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.926973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhj6\" (UniqueName: \"kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:41.927188 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:41.927069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.028417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.028317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhj6\" (UniqueName: \"kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.028417 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.028402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.029103 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.029084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.037466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.037440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhj6\" (UniqueName: \"kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6\") pod \"payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.090355 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.090329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.226981 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.226951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:54:42.230169 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:54:42.230139    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f09c29a_8447_416e_8e68_8963f0406def.slice/crio-8350cc6d148b8c84da5ebcb966f5365fd189f4b6723fa8381684bf89b41815bc WatchSource:0}: Error finding container 8350cc6d148b8c84da5ebcb966f5365fd189f4b6723fa8381684bf89b41815bc: Status 404 returned error can't find the container with id 8350cc6d148b8c84da5ebcb966f5365fd189f4b6723fa8381684bf89b41815bc
Jul 29 01:54:42.918932 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.918900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" event={"ID":"5f09c29a-8447-416e-8e68-8963f0406def","Type":"ContainerStarted","Data":"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"}
Jul 29 01:54:42.919322 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.918948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" event={"ID":"5f09c29a-8447-416e-8e68-8963f0406def","Type":"ContainerStarted","Data":"8350cc6d148b8c84da5ebcb966f5365fd189f4b6723fa8381684bf89b41815bc"}
Jul 29 01:54:42.919322 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.918979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:54:42.935386 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:42.935337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" podStartSLOduration=1.9353224789999999 podStartE2EDuration="1.935322479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:54:42.934167111 +0000 UTC m=+2670.087612484" watchObservedRunningTime="2026-07-29 01:54:42.935322479 +0000 UTC m=+2670.088767854"
Jul 29 01:54:49.060644 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:49.056773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:54:53.925631 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:54:53.925601    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:55:08.621954 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.621868    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"]
Jul 29 01:55:08.624843 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.624825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:08.627290 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.627268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 01:55:08.627453 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.627266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 01:55:08.632390 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.632069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"]
Jul 29 01:55:08.738866 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.738834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2\" (UID: \"f454ec62-4612-4ee5-b15b-578dd2db60a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:08.839523 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.839493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2\" (UID: \"f454ec62-4612-4ee5-b15b-578dd2db60a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:08.840075 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.840034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2\" (UID: \"f454ec62-4612-4ee5-b15b-578dd2db60a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:08.936715 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:08.936654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:09.064226 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:09.064200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"]
Jul 29 01:55:09.066271 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:55:09.066241    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf454ec62_4612_4ee5_b15b_578dd2db60a2.slice/crio-df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c WatchSource:0}: Error finding container df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c: Status 404 returned error can't find the container with id df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c
Jul 29 01:55:10.042532 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:10.042440    2577 generic.go:356] "Generic (PLEG): container finished" podID="f454ec62-4612-4ee5-b15b-578dd2db60a2" containerID="5e695a7980d98d1efc4657a7e9b13cf2572cec137077adbaa4b1d2676549b23b" exitCode=0
Jul 29 01:55:10.042532 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:10.042487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2" event={"ID":"f454ec62-4612-4ee5-b15b-578dd2db60a2","Type":"ContainerDied","Data":"5e695a7980d98d1efc4657a7e9b13cf2572cec137077adbaa4b1d2676549b23b"}
Jul 29 01:55:10.042532 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:10.042527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2" event={"ID":"f454ec62-4612-4ee5-b15b-578dd2db60a2","Type":"ContainerStarted","Data":"df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c"}
Jul 29 01:55:11.173210 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:11.173187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:11.259821 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:11.259789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca\") pod \"f454ec62-4612-4ee5-b15b-578dd2db60a2\" (UID: \"f454ec62-4612-4ee5-b15b-578dd2db60a2\") "
Jul 29 01:55:11.260135 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:11.260112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f454ec62-4612-4ee5-b15b-578dd2db60a2" (UID: "f454ec62-4612-4ee5-b15b-578dd2db60a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:55:11.361083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:11.361059    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454ec62-4612-4ee5-b15b-578dd2db60a2-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:12.052564 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:12.052536    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"
Jul 29 01:55:12.052758 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:12.052555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2" event={"ID":"f454ec62-4612-4ee5-b15b-578dd2db60a2","Type":"ContainerDied","Data":"df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c"}
Jul 29 01:55:12.052758 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:12.052612    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2c14c2dc2d4c9bc03fe9c85ae7b2bf2fc4c529d1eaeca68c9fc1083551fc3c"
Jul 29 01:55:24.164107 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.164071    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:55:24.164682 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.164342    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" podUID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" containerName="payload-pre-processing" containerID="cri-o://2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a" gracePeriod=30
Jul 29 01:55:24.306871 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.306834    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"]
Jul 29 01:55:24.311090 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.311065    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2f7b9a-l25q2"]
Jul 29 01:55:24.408545 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.408519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:55:24.458802 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.458728    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume\") pod \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") "
Jul 29 01:55:24.458975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.458953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gkmv\" (UniqueName: \"kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv\") pod \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\" (UID: \"8fa9c737-f2f8-4d18-b400-ec4bf9202928\") "
Jul 29 01:55:24.459122 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.459099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fa9c737-f2f8-4d18-b400-ec4bf9202928" (UID: "8fa9c737-f2f8-4d18-b400-ec4bf9202928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:55:24.459266 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.459250    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9c737-f2f8-4d18-b400-ec4bf9202928-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:24.461008 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.460985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv" (OuterVolumeSpecName: "kube-api-access-4gkmv") pod "8fa9c737-f2f8-4d18-b400-ec4bf9202928" (UID: "8fa9c737-f2f8-4d18-b400-ec4bf9202928"). InnerVolumeSpecName "kube-api-access-4gkmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:55:24.559945 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:24.559913    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkmv\" (UniqueName: \"kubernetes.io/projected/8fa9c737-f2f8-4d18-b400-ec4bf9202928-kube-api-access-4gkmv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:25.119554 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.119513    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" containerID="2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a" exitCode=0
Jul 29 01:55:25.119718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.119557    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"
Jul 29 01:55:25.119718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.119602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" event={"ID":"8fa9c737-f2f8-4d18-b400-ec4bf9202928","Type":"ContainerDied","Data":"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"}
Jul 29 01:55:25.119718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.119648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl" event={"ID":"8fa9c737-f2f8-4d18-b400-ec4bf9202928","Type":"ContainerDied","Data":"b202c3207e1a8a01a48c26a24efe2d2fd967e51d5c1692cd7ec1e1ae7c7cb3f0"}
Jul 29 01:55:25.119718 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.119672    2577 scope.go:117] "RemoveContainer" containerID="2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"
Jul 29 01:55:25.129494 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.129479    2577 scope.go:117] "RemoveContainer" containerID="2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"
Jul 29 01:55:25.129728 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:55:25.129711    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a\": container with ID starting with 2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a not found: ID does not exist" containerID="2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"
Jul 29 01:55:25.129767 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.129737    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a"} err="failed to get container status \"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a\": rpc error: code = NotFound desc = could not find container \"2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a\": container with ID starting with 2263fd3d0b433134be6bb72425d51140496339cb15fab2b8ad52827ba310fc7a not found: ID does not exist"
Jul 29 01:55:25.141857 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.141835    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:55:25.145642 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.145619    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2f7b9a-f67dfbc56-jl2jl"]
Jul 29 01:55:25.466545 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.466462    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" path="/var/lib/kubelet/pods/8fa9c737-f2f8-4d18-b400-ec4bf9202928/volumes"
Jul 29 01:55:25.466886 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:25.466760    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f454ec62-4612-4ee5-b15b-578dd2db60a2" path="/var/lib/kubelet/pods/f454ec62-4612-4ee5-b15b-578dd2db60a2/volumes"
Jul 29 01:55:26.756315 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:26.756279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:55:30.827490 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:30.827459    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:55:30.827944 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:30.827638    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" containerID="cri-o://ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0" gracePeriod=30
Jul 29 01:55:31.396980 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:31.396948    2577 patch_prober.go:28] interesting pod/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 01:55:31.397161 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:31.397014    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 01:55:36.071740 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.071713    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:55:36.154705 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154705 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154668    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154705 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154694    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrhkd\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.154962 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.154886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token\") pod \"a7081811-8a0e-40f8-8c60-ac4350ea0328\" (UID: \"a7081811-8a0e-40f8-8c60-ac4350ea0328\") "
Jul 29 01:55:36.155343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:55:36.155343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:55:36.155343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:55:36.155343 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155223    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.155556 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:55:36.155699 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.155675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:55:36.156996 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.156973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:55:36.157197 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.157173    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 01:55:36.157273 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.157251    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 01:55:36.157325 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.157276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd" (OuterVolumeSpecName: "kube-api-access-xrhkd") pod "a7081811-8a0e-40f8-8c60-ac4350ea0328" (UID: "a7081811-8a0e-40f8-8c60-ac4350ea0328"). InnerVolumeSpecName "kube-api-access-xrhkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:55:36.171944 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.171924    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerID="ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0" exitCode=0
Jul 29 01:55:36.172031 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.171969    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"
Jul 29 01:55:36.172031 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.172003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerDied","Data":"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"}
Jul 29 01:55:36.172117 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.172070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87" event={"ID":"a7081811-8a0e-40f8-8c60-ac4350ea0328","Type":"ContainerDied","Data":"379f8219741b06fd0729c44a7bdbc24ca2577ab5b9cb5562a23bf6818e047ee4"}
Jul 29 01:55:36.172117 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.172094    2577 scope.go:117] "RemoveContainer" containerID="ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"
Jul 29 01:55:36.180778 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.180762    2577 scope.go:117] "RemoveContainer" containerID="04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"
Jul 29 01:55:36.188329 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.188312    2577 scope.go:117] "RemoveContainer" containerID="ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"
Jul 29 01:55:36.188583 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:55:36.188565    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0\": container with ID starting with ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0 not found: ID does not exist" containerID="ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"
Jul 29 01:55:36.188631 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.188592    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0"} err="failed to get container status \"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0\": rpc error: code = NotFound desc = could not find container \"ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0\": container with ID starting with ab3c880064d60243a9536e25c6aa9405f33fc0625593a0a19588173b7e561be0 not found: ID does not exist"
Jul 29 01:55:36.188631 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.188609    2577 scope.go:117] "RemoveContainer" containerID="04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"
Jul 29 01:55:36.188835 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:55:36.188817    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab\": container with ID starting with 04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab not found: ID does not exist" containerID="04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"
Jul 29 01:55:36.188882 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.188841    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab"} err="failed to get container status \"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab\": rpc error: code = NotFound desc = could not find container \"04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab\": container with ID starting with 04f2ee3aedaa8dd85e8300ea83eaeb0c38b3618f5e042957d7c46f5d1d666aab not found: ID does not exist"
Jul 29 01:55:36.194013 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.193993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:55:36.200366 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.200348    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2f7b9a-openshift-default-5f89cd8554-pcw87"]
Jul 29 01:55:36.256150 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256128    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7081811-8a0e-40f8-8c60-ac4350ea0328-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256150    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrhkd\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-kube-api-access-xrhkd\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256165    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256179    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256192    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256204    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256217    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:36.256252 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:36.256229    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7081811-8a0e-40f8-8c60-ac4350ea0328-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:37.465975 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:37.465945    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" path="/var/lib/kubelet/pods/a7081811-8a0e-40f8-8c60-ac4350ea0328/volumes"
Jul 29 01:55:46.792368 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:46.792336    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:55:46.792786 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:46.792529    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" podUID="5f09c29a-8447-416e-8e68-8963f0406def" containerName="payload-pre-processing" containerID="cri-o://d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d" gracePeriod=30
Jul 29 01:55:47.040019 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.039992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:55:47.141745 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.141720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume\") pod \"5f09c29a-8447-416e-8e68-8963f0406def\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") "
Jul 29 01:55:47.141880 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.141772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhj6\" (UniqueName: \"kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6\") pod \"5f09c29a-8447-416e-8e68-8963f0406def\" (UID: \"5f09c29a-8447-416e-8e68-8963f0406def\") "
Jul 29 01:55:47.142149 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.142121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f09c29a-8447-416e-8e68-8963f0406def" (UID: "5f09c29a-8447-416e-8e68-8963f0406def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 01:55:47.143774 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.143756    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6" (OuterVolumeSpecName: "kube-api-access-2dhj6") pod "5f09c29a-8447-416e-8e68-8963f0406def" (UID: "5f09c29a-8447-416e-8e68-8963f0406def"). InnerVolumeSpecName "kube-api-access-2dhj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 01:55:47.225843 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.225806    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f09c29a-8447-416e-8e68-8963f0406def" containerID="d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d" exitCode=0
Jul 29 01:55:47.225956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.225844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"
Jul 29 01:55:47.225956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.225882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" event={"ID":"5f09c29a-8447-416e-8e68-8963f0406def","Type":"ContainerDied","Data":"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"}
Jul 29 01:55:47.225956 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.225937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7" event={"ID":"5f09c29a-8447-416e-8e68-8963f0406def","Type":"ContainerDied","Data":"8350cc6d148b8c84da5ebcb966f5365fd189f4b6723fa8381684bf89b41815bc"}
Jul 29 01:55:47.226136 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.225961    2577 scope.go:117] "RemoveContainer" containerID="d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"
Jul 29 01:55:47.234802 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.234782    2577 scope.go:117] "RemoveContainer" containerID="d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"
Jul 29 01:55:47.235079 ip-10-0-136-251 kubenswrapper[2577]: E0729 01:55:47.235056    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d\": container with ID starting with d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d not found: ID does not exist" containerID="d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"
Jul 29 01:55:47.235134 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.235089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d"} err="failed to get container status \"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d\": rpc error: code = NotFound desc = could not find container \"d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d\": container with ID starting with d8af7c3f433f48108f0cf78819811c902aa0792151cdcf186f61f6690166551d not found: ID does not exist"
Jul 29 01:55:47.242475 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.242458    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhj6\" (UniqueName: \"kubernetes.io/projected/5f09c29a-8447-416e-8e68-8963f0406def-kube-api-access-2dhj6\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:47.242531 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.242477    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09c29a-8447-416e-8e68-8963f0406def-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 01:55:47.247441 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.247421    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:55:47.251635 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.251616    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9132e3-864cf88cd4-ptzt7"]
Jul 29 01:55:47.465745 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:47.465677    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f09c29a-8447-416e-8e68-8963f0406def" path="/var/lib/kubelet/pods/5f09c29a-8447-416e-8e68-8963f0406def/volumes"
Jul 29 01:55:51.448949 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:51.448913    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:55:54.217131 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217096    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" containerName="payload-pre-processing"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217414    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" containerName="payload-pre-processing"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217429    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f09c29a-8447-416e-8e68-8963f0406def" containerName="payload-pre-processing"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217435    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f09c29a-8447-416e-8e68-8963f0406def" containerName="payload-pre-processing"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217463    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217468    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f454ec62-4612-4ee5-b15b-578dd2db60a2" containerName="revoke-keys"
Jul 29 01:55:54.217492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217472    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f454ec62-4612-4ee5-b15b-578dd2db60a2" containerName="revoke-keys"
Jul 29 01:55:54.217785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217526    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f09c29a-8447-416e-8e68-8963f0406def" containerName="payload-pre-processing"
Jul 29 01:55:54.217785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217534    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9c737-f2f8-4d18-b400-ec4bf9202928" containerName="payload-pre-processing"
Jul 29 01:55:54.217785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217540    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:54.217785 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.217545    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f454ec62-4612-4ee5-b15b-578dd2db60a2" containerName="revoke-keys"
Jul 29 01:55:54.220462 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.220443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.223289 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.223181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-27cc80-openshift-default-dockercfg-pmm8g\""
Jul 29 01:55:54.223533 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.223306    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 01:55:54.233225 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.233203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 01:55:54.394101 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394101 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb49s\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394282 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.394466 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.394312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495130 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb49s\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495397 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495711 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495711 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495801 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495801 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.495955 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.495932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.497482 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.497455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.497675 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.497658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.504549 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.504517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb49s\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.504676 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.504604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token\") pod \"e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") " pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.530647 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.530624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:54.680120 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.680097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 01:55:54.682504 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:55:54.682479    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5773bece_b2cc_4b29_bad2_9de1db64cf6a.slice/crio-446301f8dd5e511fdc61b2e3219f402b0e67fe19cf4eb7468e2aa55aa248c21f WatchSource:0}: Error finding container 446301f8dd5e511fdc61b2e3219f402b0e67fe19cf4eb7468e2aa55aa248c21f: Status 404 returned error can't find the container with id 446301f8dd5e511fdc61b2e3219f402b0e67fe19cf4eb7468e2aa55aa248c21f
Jul 29 01:55:54.684946 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.684913    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:55:54.685024 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.684984    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:55:54.685024 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:54.685010    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:55:55.262394 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:55.262360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerStarted","Data":"3e9825a946d62f13fdc1dde8f3b85ac9d17da8d2e7ca8d3b92fe599158a5ab07"}
Jul 29 01:55:55.262852 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:55.262415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerStarted","Data":"446301f8dd5e511fdc61b2e3219f402b0e67fe19cf4eb7468e2aa55aa248c21f"}
Jul 29 01:55:55.285374 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:55.285328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podStartSLOduration=1.285313502 podStartE2EDuration="1.285313502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:55:55.283217073 +0000 UTC m=+2742.436662450" watchObservedRunningTime="2026-07-29 01:55:55.285313502 +0000 UTC m=+2742.438758877"
Jul 29 01:55:55.531467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:55.531393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:55.536201 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:55.536179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:56.267765 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:56.267734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:56.268575 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:56.268553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:55:58.181444 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.181409    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 01:55:58.181850 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.181837    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7081811-8a0e-40f8-8c60-ac4350ea0328" containerName="istio-proxy"
Jul 29 01:55:58.184604 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.184571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.188289 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.188265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-27cc80-dockercfg-l6dth\""
Jul 29 01:55:58.188667 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.188635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-27cc80\""
Jul 29 01:55:58.196595 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.196560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 01:55:58.200051 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.200009    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 01:55:58.210836 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.210813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.215001 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.214979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 01:55:58.328359 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.328330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbcr\" (UniqueName: \"kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.328359 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.328364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.328549 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.328412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.328549 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.328455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skc7q\" (UniqueName: \"kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.429468 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.429441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbcr\" (UniqueName: \"kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.429645 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.429474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.429645 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.429521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.429645 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.429569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skc7q\" (UniqueName: \"kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.430346 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.430320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.430346 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.430332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.440335 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.440274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbcr\" (UniqueName: \"kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr\") pod \"payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.440416 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.440353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skc7q\" (UniqueName: \"kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q\") pod \"payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.498405 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.498381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:58.521967 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.521948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:58.644291 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.644175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 01:55:58.646947 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:55:58.646914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bee2796_45f3_46a3_af75_0f3a8bedc5ae.slice/crio-399be020511a36f41f4ab491ff7e9d761931f6f4a3f8e5f1776161c75f09bc17 WatchSource:0}: Error finding container 399be020511a36f41f4ab491ff7e9d761931f6f4a3f8e5f1776161c75f09bc17: Status 404 returned error can't find the container with id 399be020511a36f41f4ab491ff7e9d761931f6f4a3f8e5f1776161c75f09bc17
Jul 29 01:55:58.657331 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:58.657314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 01:55:58.660481 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:55:58.660456    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e30741f_4aed_4931_9a1f_cc02d6e2bfe5.slice/crio-a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee WatchSource:0}: Error finding container a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee: Status 404 returned error can't find the container with id a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee
Jul 29 01:55:59.287503 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.287470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" event={"ID":"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5","Type":"ContainerStarted","Data":"cf702df7a0222f7efa8aa84d0ae71a5933ad3fbd7f7b0b4be318ff764b898de2"}
Jul 29 01:55:59.287946 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.287521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" event={"ID":"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5","Type":"ContainerStarted","Data":"a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee"}
Jul 29 01:55:59.287946 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.287559    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:55:59.289167 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.289146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" event={"ID":"4bee2796-45f3-46a3-af75-0f3a8bedc5ae","Type":"ContainerStarted","Data":"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"}
Jul 29 01:55:59.289250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.289181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" event={"ID":"4bee2796-45f3-46a3-af75-0f3a8bedc5ae","Type":"ContainerStarted","Data":"399be020511a36f41f4ab491ff7e9d761931f6f4a3f8e5f1776161c75f09bc17"}
Jul 29 01:55:59.289250 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.289209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:55:59.308638 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.308589    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" podStartSLOduration=1.308578029 podStartE2EDuration="1.308578029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:55:59.306501108 +0000 UTC m=+2746.459946484" watchObservedRunningTime="2026-07-29 01:55:59.308578029 +0000 UTC m=+2746.462023404"
Jul 29 01:55:59.323238 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:55:59.323194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" podStartSLOduration=1.323182143 podStartE2EDuration="1.323182143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:55:59.322142202 +0000 UTC m=+2746.475587578" watchObservedRunningTime="2026-07-29 01:55:59.323182143 +0000 UTC m=+2746.476627519"
Jul 29 01:56:01.660255 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:01.660222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:56:10.297096 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.297065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 01:56:10.297527 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.297143    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 01:56:10.344602 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.344571    2577 generic.go:356] "Generic (PLEG): container finished" podID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerID="3e9825a946d62f13fdc1dde8f3b85ac9d17da8d2e7ca8d3b92fe599158a5ab07" exitCode=0
Jul 29 01:56:10.344758 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.344614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerDied","Data":"3e9825a946d62f13fdc1dde8f3b85ac9d17da8d2e7ca8d3b92fe599158a5ab07"}
Jul 29 01:56:10.344931 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.344915    2577 scope.go:117] "RemoveContainer" containerID="3e9825a946d62f13fdc1dde8f3b85ac9d17da8d2e7ca8d3b92fe599158a5ab07"
Jul 29 01:56:10.345981 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.345944    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:56:10.346111 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.346061    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:56:10.346158 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:10.346119    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 01:56:11.268536 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:11.268493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:56:11.369376 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:11.369333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerStarted","Data":"1ca3511b1292d712cfb25679317068c0d34fa141fcc1f99ecd2f5b3a951b6bef"}
Jul 29 01:56:11.531272 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:11.531178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:56:11.532341 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:11.532308    2577 patch_prober.go:28] interesting pod/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 29 01:56:11.532467 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:11.532356    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 29 01:56:12.530940 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:12.530906    2577 patch_prober.go:28] interesting pod/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 29 01:56:12.531361 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:12.530955    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 29 01:56:13.531205 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:13.531167    2577 patch_prober.go:28] interesting pod/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 29 01:56:13.531618 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:13.531213    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 29 01:56:14.531787 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:14.531749    2577 patch_prober.go:28] interesting pod/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 29 01:56:14.532203 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:14.531826    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 29 01:56:15.535677 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:15.535646    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:56:15.536058 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:15.535934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:56:15.536681 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:15.536663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 01:56:26.869825 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.869783    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 01:56:26.873203 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.873179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:26.876423 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.876401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-a7bf0d-dockercfg-vhpm9\""
Jul 29 01:56:26.876541 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.876424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-a7bf0d\""
Jul 29 01:56:26.883567 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.883549    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 01:56:26.886712 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.886692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 01:56:26.886813 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.886771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:26.898483 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.898462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 01:56:26.967327 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.967305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:26.967469 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.967343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mwv\" (UniqueName: \"kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:26.967469 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.967382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:26.967469 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:26.967440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ppg\" (UniqueName: \"kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.068597 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.068564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.068756 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.068603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mwv\" (UniqueName: \"kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.068756 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.068627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.068756 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.068660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ppg\" (UniqueName: \"kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.069610 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.069586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.069662 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.069589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.079735 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.079708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ppg\" (UniqueName: \"kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg\") pod \"payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.080020 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.080004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mwv\" (UniqueName: \"kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv\") pod \"payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.184242 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.184151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:27.196333 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.196309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.360670 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.360643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 01:56:27.363295 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:56:27.363260    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b2036c_0206_456d_a82e_0241db1f6e6c.slice/crio-6b3cae9bbb96d25d79aecc09e41b98cf4c51283a513d0a41abf83f074e525aaa WatchSource:0}: Error finding container 6b3cae9bbb96d25d79aecc09e41b98cf4c51283a513d0a41abf83f074e525aaa: Status 404 returned error can't find the container with id 6b3cae9bbb96d25d79aecc09e41b98cf4c51283a513d0a41abf83f074e525aaa
Jul 29 01:56:27.444571 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.444502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" event={"ID":"51b2036c-0206-456d-a82e-0241db1f6e6c","Type":"ContainerStarted","Data":"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"}
Jul 29 01:56:27.444571 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.444558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" event={"ID":"51b2036c-0206-456d-a82e-0241db1f6e6c","Type":"ContainerStarted","Data":"6b3cae9bbb96d25d79aecc09e41b98cf4c51283a513d0a41abf83f074e525aaa"}
Jul 29 01:56:27.444733 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.444612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:27.464535 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.464495    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" podStartSLOduration=1.464481136 podStartE2EDuration="1.464481136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:56:27.462239502 +0000 UTC m=+2774.615684873" watchObservedRunningTime="2026-07-29 01:56:27.464481136 +0000 UTC m=+2774.617926512"
Jul 29 01:56:27.544852 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:27.544827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 01:56:27.546735 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:56:27.546707    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58892a28_c854_4bf3_abda_4a580960591c.slice/crio-e78be7011134194755a9a3dbff7faac867d3922cbba9e4e2e5e7ffa336b45d4d WatchSource:0}: Error finding container e78be7011134194755a9a3dbff7faac867d3922cbba9e4e2e5e7ffa336b45d4d: Status 404 returned error can't find the container with id e78be7011134194755a9a3dbff7faac867d3922cbba9e4e2e5e7ffa336b45d4d
Jul 29 01:56:28.451202 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:28.451167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" event={"ID":"58892a28-c854-4bf3-abda-4a580960591c","Type":"ContainerStarted","Data":"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"}
Jul 29 01:56:28.451579 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:28.451229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" event={"ID":"58892a28-c854-4bf3-abda-4a580960591c","Type":"ContainerStarted","Data":"e78be7011134194755a9a3dbff7faac867d3922cbba9e4e2e5e7ffa336b45d4d"}
Jul 29 01:56:28.468133 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:28.468088    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" podStartSLOduration=2.468073258 podStartE2EDuration="2.468073258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:56:28.466484583 +0000 UTC m=+2775.619929961" watchObservedRunningTime="2026-07-29 01:56:28.468073258 +0000 UTC m=+2775.621518633"
Jul 29 01:56:29.466635 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:29.466559    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:34.058643 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:34.058600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:56:38.453927 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:38.453896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 01:56:40.468614 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:40.468583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 01:56:58.652543 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.652508    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 01:56:58.655963 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.655946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.659918 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.659894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"default-dockercfg-kmnhz\""
Jul 29 01:56:58.659918 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.659910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"openshift-service-ca.crt\""
Jul 29 01:56:58.660143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.659960    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"kube-root-ca.crt\""
Jul 29 01:56:58.660143 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.659996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"auth-test-model-27cc80-kserve-self-signed-certs\""
Jul 29 01:56:58.665842 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.665823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 01:56:58.813381 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.813573 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.813573 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.813573 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4sph\" (UniqueName: \"kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.813695 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.813695 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.813636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.914980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915083 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915374 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sph\" (UniqueName: \"kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915374 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915374 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915521 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915521 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.915594 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.915522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.917456 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.917428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.917624 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.917607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.923440 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.923413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4sph\" (UniqueName: \"kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph\") pod \"auth-test-model-27cc80-kserve-7ddbb5c775-prxdn\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") " pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:58.966459 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:58.966429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:56:59.095595 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:59.095571    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 01:56:59.098283 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:56:59.098257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6d28374_e77b_40bb_89d4_ae86be0e94b8.slice/crio-b816af27d4d9959ba263675c2b0af391ec375958e657adc1365c8069d3c582fa WatchSource:0}: Error finding container b816af27d4d9959ba263675c2b0af391ec375958e657adc1365c8069d3c582fa: Status 404 returned error can't find the container with id b816af27d4d9959ba263675c2b0af391ec375958e657adc1365c8069d3c582fa
Jul 29 01:56:59.598419 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:59.598384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" event={"ID":"e6d28374-e77b-40bb-89d4-ae86be0e94b8","Type":"ContainerStarted","Data":"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"}
Jul 29 01:56:59.598620 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:59.598426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" event={"ID":"e6d28374-e77b-40bb-89d4-ae86be0e94b8","Type":"ContainerStarted","Data":"b816af27d4d9959ba263675c2b0af391ec375958e657adc1365c8069d3c582fa"}
Jul 29 01:56:59.617137 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:56:59.617091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" podStartSLOduration=1.617077176 podStartE2EDuration="1.617077176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:56:59.615338148 +0000 UTC m=+2806.768783524" watchObservedRunningTime="2026-07-29 01:56:59.617077176 +0000 UTC m=+2806.770522552"
Jul 29 01:57:01.062784 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:01.062752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:57:08.967207 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:08.967173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:57:08.967665 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:08.967237    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:57:08.979646 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:08.979617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:57:09.661510 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:09.661481    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 01:57:21.966762 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.966721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 01:57:21.970203 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.970181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:21.974628 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.974603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"kube-root-ca.crt\""
Jul 29 01:57:21.974756 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.974713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"auth-test-model-a7bf0d-kserve-self-signed-certs\""
Jul 29 01:57:21.974835 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.974755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"openshift-service-ca.crt\""
Jul 29 01:57:21.974835 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.974824    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"default-dockercfg-78mbk\""
Jul 29 01:57:21.983410 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:21.983386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 01:57:22.009386 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7td\" (UniqueName: \"kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.009492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.009492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.009492 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.009617 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.009617 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.009530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.110760 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.110718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.110760 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.110765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.110965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.110797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.110965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.110864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7td\" (UniqueName: \"kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.110965 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.110940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.111165 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.111085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.111246 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.111156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.111314 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.111281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.111441 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.111424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.113053 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.113022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.113309 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.113292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.120010 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.119984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7td\" (UniqueName: \"kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td\") pod \"auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") " pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.281600 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.281511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:22.413178 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.413146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 01:57:22.415173 ip-10-0-136-251 kubenswrapper[2577]: W0729 01:57:22.415147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode503042f_1b62_45c1_bfc5_6875f5ab6bb3.slice/crio-6b29203b900ebabbf90149cadf207fbaaaf3fc2c46d1e28f2282a1fce72d0cb7 WatchSource:0}: Error finding container 6b29203b900ebabbf90149cadf207fbaaaf3fc2c46d1e28f2282a1fce72d0cb7: Status 404 returned error can't find the container with id 6b29203b900ebabbf90149cadf207fbaaaf3fc2c46d1e28f2282a1fce72d0cb7
Jul 29 01:57:22.714202 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.714165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" event={"ID":"e503042f-1b62-45c1-bfc5-6875f5ab6bb3","Type":"ContainerStarted","Data":"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"}
Jul 29 01:57:22.714202 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.714207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" event={"ID":"e503042f-1b62-45c1-bfc5-6875f5ab6bb3","Type":"ContainerStarted","Data":"6b29203b900ebabbf90149cadf207fbaaaf3fc2c46d1e28f2282a1fce72d0cb7"}
Jul 29 01:57:22.733476 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:22.733402    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" podStartSLOduration=1.7333870820000001 podStartE2EDuration="1.733387082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:57:22.73226549 +0000 UTC m=+2829.885710877" watchObservedRunningTime="2026-07-29 01:57:22.733387082 +0000 UTC m=+2829.886832506"
Jul 29 01:57:25.856233 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:25.856201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:57:32.281892 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:32.281853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:32.282474 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:32.282008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:32.294478 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:32.294457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:32.771863 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:32.771830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 01:57:58.055792 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:57:58.055751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:58:22.269584 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:58:22.269502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:58:36.058387 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:58:36.058346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:58:49.254623 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:58:49.254590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:59:05.966764 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:59:05.966726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:59:24.658462 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:59:24.658426    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 01:59:46.684853 ip-10-0-136-251 kubenswrapper[2577]: I0729 01:59:46.684822    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:00:00.984351 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:00:00.984316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:00:12.292383 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:00:12.292346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:00:13.647205 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:00:13.647173    2577 scope.go:117] "RemoveContainer" containerID="3770431e46a439a5b704cf126df31f3947b8e2a0911b1a3b4b146d93a3aaa5ec"
Jul 29 02:00:27.853615 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:00:27.853578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:00:42.372773 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:00:42.372732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:01:03.559752 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:03.559718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:01:13.668421 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:13.668385    2577 scope.go:117] "RemoveContainer" containerID="5e695a7980d98d1efc4657a7e9b13cf2572cec137077adbaa4b1d2676549b23b"
Jul 29 02:01:21.663990 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:21.663953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:01:43.758733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:43.758694    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:01:48.742706 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:48.742670    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 02:01:49.661181 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:49.661150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 02:01:57.121655 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.121620    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:01:57.124968 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.124950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.134174 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.134155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"sub-test-model-27cc80-kserve-self-signed-certs\""
Jul 29 02:01:57.180536 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.180511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:01:57.283586 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwr7f\" (UniqueName: \"kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.283586 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.283742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.283742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.283742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.283742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.283702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384657 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwr7f\" (UniqueName: \"kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384657 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384657 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384855 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384855 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.384855 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.384821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.385136 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.385113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.385212 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.385132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.385212 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.385183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.386803 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.386775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.387169 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.387150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.392812 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.392788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwr7f\" (UniqueName: \"kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f\") pod \"sub-test-model-27cc80-kserve-66c5cb5669-xk4zf\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") " pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.433685 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.433661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:01:57.564763 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:57.564738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:01:58.012644 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:58.012607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" event={"ID":"88723e4d-7023-42c6-9dd3-8bf50cfcecd2","Type":"ContainerStarted","Data":"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"}
Jul 29 02:01:58.012644 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:58.012649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" event={"ID":"88723e4d-7023-42c6-9dd3-8bf50cfcecd2","Type":"ContainerStarted","Data":"dc42b851c9330890df3fe0a09350f66bc9a4d6e875d0a8dd80e18c65e1626334"}
Jul 29 02:01:58.031849 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:01:58.031797    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" podStartSLOduration=1.031779032 podStartE2EDuration="1.031779032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:01:58.029457722 +0000 UTC m=+3105.182903098" watchObservedRunningTime="2026-07-29 02:01:58.031779032 +0000 UTC m=+3105.185224460"
Jul 29 02:02:00.986157 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:00.986121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:02:03.761452 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:03.761396    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" podUID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" containerName="main" containerID="cri-o://1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396" gracePeriod=45
Jul 29 02:02:04.014216 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.014158    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 02:02:04.041886 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.041855    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" containerID="1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396" exitCode=0
Jul 29 02:02:04.042000 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.041887    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"
Jul 29 02:02:04.042000 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.041924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" event={"ID":"e6d28374-e77b-40bb-89d4-ae86be0e94b8","Type":"ContainerDied","Data":"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"}
Jul 29 02:02:04.042000 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.041981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn" event={"ID":"e6d28374-e77b-40bb-89d4-ae86be0e94b8","Type":"ContainerDied","Data":"b816af27d4d9959ba263675c2b0af391ec375958e657adc1365c8069d3c582fa"}
Jul 29 02:02:04.042000 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.042001    2577 scope.go:117] "RemoveContainer" containerID="1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"
Jul 29 02:02:04.050215 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.050191    2577 scope.go:117] "RemoveContainer" containerID="1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"
Jul 29 02:02:04.050465 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:02:04.050446    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396\": container with ID starting with 1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396 not found: ID does not exist" containerID="1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"
Jul 29 02:02:04.050553 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.050472    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396"} err="failed to get container status \"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396\": rpc error: code = NotFound desc = could not find container \"1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396\": container with ID starting with 1b3df2e69e9c3ce08fec92cb5d8e936c06ff4296e9b850f106a45be7d1b05396 not found: ID does not exist"
Jul 29 02:02:04.149866 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.149845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150019 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.149871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150019 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.149913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150019 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.149945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150019 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.149990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4sph\" (UniqueName: \"kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150280 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs\") pod \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\" (UID: \"e6d28374-e77b-40bb-89d4-ae86be0e94b8\") "
Jul 29 02:02:04.150280 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:04.150280 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150176    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:04.150280 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home" (OuterVolumeSpecName: "home") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:04.150484 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150398    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.150484 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150412    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.150484 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.150424    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.152214 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.152189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:02:04.152214 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.152194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm" (OuterVolumeSpecName: "dshm") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:04.152340 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.152299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph" (OuterVolumeSpecName: "kube-api-access-c4sph") pod "e6d28374-e77b-40bb-89d4-ae86be0e94b8" (UID: "e6d28374-e77b-40bb-89d4-ae86be0e94b8"). InnerVolumeSpecName "kube-api-access-c4sph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:02:04.251420 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.251394    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d28374-e77b-40bb-89d4-ae86be0e94b8-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.251420 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.251415    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d28374-e77b-40bb-89d4-ae86be0e94b8-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.251420 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.251424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4sph\" (UniqueName: \"kubernetes.io/projected/e6d28374-e77b-40bb-89d4-ae86be0e94b8-kube-api-access-c4sph\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:04.363462 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.363428    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 02:02:04.367132 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.367109    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/auth-test-model-27cc80-kserve-7ddbb5c775-prxdn"]
Jul 29 02:02:04.680488 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.680403    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" podUID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" containerName="main" containerID="cri-o://e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253" gracePeriod=45
Jul 29 02:02:04.925148 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:04.925127    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 02:02:05.049321 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.049220    2577 generic.go:356] "Generic (PLEG): container finished" podID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" containerID="e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253" exitCode=0
Jul 29 02:02:05.049321 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.049256    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"
Jul 29 02:02:05.049321 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.049281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" event={"ID":"e503042f-1b62-45c1-bfc5-6875f5ab6bb3","Type":"ContainerDied","Data":"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"}
Jul 29 02:02:05.049321 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.049318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs" event={"ID":"e503042f-1b62-45c1-bfc5-6875f5ab6bb3","Type":"ContainerDied","Data":"6b29203b900ebabbf90149cadf207fbaaaf3fc2c46d1e28f2282a1fce72d0cb7"}
Jul 29 02:02:05.049589 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.049341    2577 scope.go:117] "RemoveContainer" containerID="e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"
Jul 29 02:02:05.057473 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.057458    2577 scope.go:117] "RemoveContainer" containerID="e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"
Jul 29 02:02:05.057726 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:02:05.057708    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253\": container with ID starting with e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253 not found: ID does not exist" containerID="e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"
Jul 29 02:02:05.057795 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.057733    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253"} err="failed to get container status \"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253\": rpc error: code = NotFound desc = could not find container \"e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253\": container with ID starting with e2ef2d99e8a76b1b7b8d1bba6b413a1dd9e4d9c487cfc0945e91713ac0bc5253 not found: ID does not exist"
Jul 29 02:02:05.060086 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr7td\" (UniqueName: \"kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060148 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060148 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060132    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060255 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060310 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060359 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home\") pod \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\" (UID: \"e503042f-1b62-45c1-bfc5-6875f5ab6bb3\") "
Jul 29 02:02:05.060415 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:05.060475 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:05.060665 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060632    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home" (OuterVolumeSpecName: "home") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:05.060748 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060696    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.060748 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.060714    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.062327 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.062298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td" (OuterVolumeSpecName: "kube-api-access-gr7td") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "kube-api-access-gr7td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:02:05.062327 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.062307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:02:05.062462 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.062334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm" (OuterVolumeSpecName: "dshm") pod "e503042f-1b62-45c1-bfc5-6875f5ab6bb3" (UID: "e503042f-1b62-45c1-bfc5-6875f5ab6bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:02:05.161510 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.161490    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.161510 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.161509    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.161643 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.161518    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7td\" (UniqueName: \"kubernetes.io/projected/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-kube-api-access-gr7td\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.161643 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.161526    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e503042f-1b62-45c1-bfc5-6875f5ab6bb3-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:02:05.372948 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.372918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 02:02:05.377156 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.377135    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/auth-test-model-a7bf0d-kserve-56cd488c4c-pl7hs"]
Jul 29 02:02:05.471157 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.471126    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" path="/var/lib/kubelet/pods/e503042f-1b62-45c1-bfc5-6875f5ab6bb3/volumes"
Jul 29 02:02:05.471526 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:05.471512    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" path="/var/lib/kubelet/pods/e6d28374-e77b-40bb-89d4-ae86be0e94b8/volumes"
Jul 29 02:02:07.434349 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:07.434318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:02:07.434750 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:07.434383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:02:07.446789 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:07.446765    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:02:08.075176 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:08.075149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:02:21.565329 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565296    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" containerName="main"
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565649    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" containerName="main"
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565670    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" containerName="main"
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565675    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" containerName="main"
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565736    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d28374-e77b-40bb-89d4-ae86be0e94b8" containerName="main"
Jul 29 02:02:21.565770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.565744    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e503042f-1b62-45c1-bfc5-6875f5ab6bb3" containerName="main"
Jul 29 02:02:21.570242 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.570222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.574215 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.574191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"kube-root-ca.crt\""
Jul 29 02:02:21.574332 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.574218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"default-dockercfg-78mbk\""
Jul 29 02:02:21.574332 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.574240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"openshift-service-ca.crt\""
Jul 29 02:02:21.574332 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.574192    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"sub-test-model-a7bf0d-kserve-self-signed-certs\""
Jul 29 02:02:21.579824 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.579800    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:02:21.582274 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.582377 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.582377 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.582493 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.582493 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.582493 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.582465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbll\" (UniqueName: \"kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683119 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683225 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683225 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683225 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683225 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbll\" (UniqueName: \"kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683453 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683453 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683576 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.683629 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.683599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.685322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.685304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.685554 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.685539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.694219 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.694198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbll\" (UniqueName: \"kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll\") pod \"sub-test-model-a7bf0d-kserve-84959fd547-7mbvq\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") " pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:21.879854 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:21.879796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:22.006036 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:22.005921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:02:22.008665 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:02:22.008638    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b77077_7740_44a1_88a3_cc5c83f0cce7.slice/crio-e674c94ac2b1c1c696ed53e8dfa876ac7c2d76ecdab09672052dd76bd800558b WatchSource:0}: Error finding container e674c94ac2b1c1c696ed53e8dfa876ac7c2d76ecdab09672052dd76bd800558b: Status 404 returned error can't find the container with id e674c94ac2b1c1c696ed53e8dfa876ac7c2d76ecdab09672052dd76bd800558b
Jul 29 02:02:22.128119 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:22.128088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" event={"ID":"91b77077-7740-44a1-88a3-cc5c83f0cce7","Type":"ContainerStarted","Data":"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"}
Jul 29 02:02:22.128256 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:22.128128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" event={"ID":"91b77077-7740-44a1-88a3-cc5c83f0cce7","Type":"ContainerStarted","Data":"e674c94ac2b1c1c696ed53e8dfa876ac7c2d76ecdab09672052dd76bd800558b"}
Jul 29 02:02:22.147536 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:22.147442    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" podStartSLOduration=1.147429073 podStartE2EDuration="1.147429073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:02:22.145788025 +0000 UTC m=+3129.299233424" watchObservedRunningTime="2026-07-29 02:02:22.147429073 +0000 UTC m=+3129.300874448"
Jul 29 02:02:25.248967 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:25.248929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:02:31.880172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:31.880133    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:31.880654 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:31.880251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:31.892622 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:31.892600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:32.186712 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:32.186651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:02:57.959600 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:02:57.959565    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:03:15.858843 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:15.858807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:03:21.605121 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:21.605090    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:03:22.475734 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:22.475701    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:03:32.190401 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.190366    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:03:32.193751 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.193734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.199282 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.199258    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-27cc80\"/\"rate-test-model-27cc80-kserve-self-signed-certs\""
Jul 29 02:03:32.209100 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.209075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:03:32.265872 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.265848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.265989 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.265878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.265989 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.265900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.265989 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.265929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh76j\" (UniqueName: \"kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.266174 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.265998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.266174 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.266014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366664 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366782 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366828 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366865 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366899 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.366950 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.366920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh76j\" (UniqueName: \"kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.367081 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.367032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.367176 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.367089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.367306 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.367283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.368959 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.368939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.369215 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.369196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.376300 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.376280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh76j\" (UniqueName: \"kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j\") pod \"rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") " pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.502295 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.502222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:32.637662 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:32.637625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:03:32.641454 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:03:32.641430    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c96dc16_1784_4d8d_8e93_d428846f08af.slice/crio-be5d7550a383952da5433a9a93cdd0ee4d7fba0a591223b4026ebc12687b1c0c WatchSource:0}: Error finding container be5d7550a383952da5433a9a93cdd0ee4d7fba0a591223b4026ebc12687b1c0c: Status 404 returned error can't find the container with id be5d7550a383952da5433a9a93cdd0ee4d7fba0a591223b4026ebc12687b1c0c
Jul 29 02:03:33.459164 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:33.459120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" event={"ID":"2c96dc16-1784-4d8d-8e93-d428846f08af","Type":"ContainerStarted","Data":"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"}
Jul 29 02:03:33.459164 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:33.459166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" event={"ID":"2c96dc16-1784-4d8d-8e93-d428846f08af","Type":"ContainerStarted","Data":"be5d7550a383952da5433a9a93cdd0ee4d7fba0a591223b4026ebc12687b1c0c"}
Jul 29 02:03:33.483369 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:33.483322    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" podStartSLOduration=1.483307706 podStartE2EDuration="1.483307706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:03:33.481680362 +0000 UTC m=+3200.635125737" watchObservedRunningTime="2026-07-29 02:03:33.483307706 +0000 UTC m=+3200.636753082"
Jul 29 02:03:35.249915 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:35.249876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:03:36.623363 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:36.623324    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" podUID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" containerName="main" containerID="cri-o://6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724" gracePeriod=45
Jul 29 02:03:36.879032 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:36.878971    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:03:37.008127 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008270 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008270 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwr7f\" (UniqueName: \"kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008270 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008408 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008408 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir\") pod \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\" (UID: \"88723e4d-7023-42c6-9dd3-8bf50cfcecd2\") "
Jul 29 02:03:37.008537 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home" (OuterVolumeSpecName: "home") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.008537 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.008648 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.008708 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008679    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.008708 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008704    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.008780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.008720    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.010308 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.010277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm" (OuterVolumeSpecName: "dshm") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.010411 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.010335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:03:37.010411 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.010347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f" (OuterVolumeSpecName: "kube-api-access-qwr7f") pod "88723e4d-7023-42c6-9dd3-8bf50cfcecd2" (UID: "88723e4d-7023-42c6-9dd3-8bf50cfcecd2"). InnerVolumeSpecName "kube-api-access-qwr7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:03:37.110058 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.110012    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwr7f\" (UniqueName: \"kubernetes.io/projected/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-kube-api-access-qwr7f\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.110058 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.110054    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.110058 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.110065    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88723e4d-7023-42c6-9dd3-8bf50cfcecd2-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.479013 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.478987    2577 generic.go:356] "Generic (PLEG): container finished" podID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" containerID="6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724" exitCode=0
Jul 29 02:03:37.479175 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.479028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"
Jul 29 02:03:37.479175 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.479097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" event={"ID":"88723e4d-7023-42c6-9dd3-8bf50cfcecd2","Type":"ContainerDied","Data":"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"}
Jul 29 02:03:37.479175 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.479140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf" event={"ID":"88723e4d-7023-42c6-9dd3-8bf50cfcecd2","Type":"ContainerDied","Data":"dc42b851c9330890df3fe0a09350f66bc9a4d6e875d0a8dd80e18c65e1626334"}
Jul 29 02:03:37.479175 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.479156    2577 scope.go:117] "RemoveContainer" containerID="6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"
Jul 29 02:03:37.487372 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.487353    2577 scope.go:117] "RemoveContainer" containerID="6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"
Jul 29 02:03:37.487613 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:03:37.487594    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724\": container with ID starting with 6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724 not found: ID does not exist" containerID="6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"
Jul 29 02:03:37.487667 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.487619    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724"} err="failed to get container status \"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724\": rpc error: code = NotFound desc = could not find container \"6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724\": container with ID starting with 6a4c658f161bd1adbf8c709a50d10833e04f77158e1f06cadb13420a3a786724 not found: ID does not exist"
Jul 29 02:03:37.493807 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.493784    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" podUID="91b77077-7740-44a1-88a3-cc5c83f0cce7" containerName="main" containerID="cri-o://604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2" gracePeriod=45
Jul 29 02:03:37.497382 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.497360    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:03:37.500992 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.500972    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/sub-test-model-27cc80-kserve-66c5cb5669-xk4zf"]
Jul 29 02:03:37.748537 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.748515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:03:37.815322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815438 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815438 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815363    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815438 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815541 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtbll\" (UniqueName: \"kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815541 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache\") pod \"91b77077-7740-44a1-88a3-cc5c83f0cce7\" (UID: \"91b77077-7740-44a1-88a3-cc5c83f0cce7\") "
Jul 29 02:03:37.815623 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.815623 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home" (OuterVolumeSpecName: "home") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.815774 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815731    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.815831 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815786    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.815831 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.815806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.817506 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.817481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm" (OuterVolumeSpecName: "dshm") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:03:37.817506 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.817494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll" (OuterVolumeSpecName: "kube-api-access-vtbll") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "kube-api-access-vtbll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:03:37.817648 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.817537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91b77077-7740-44a1-88a3-cc5c83f0cce7" (UID: "91b77077-7740-44a1-88a3-cc5c83f0cce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:03:37.916170 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.916145    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.916170 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.916169    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b77077-7740-44a1-88a3-cc5c83f0cce7-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.916324 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.916182    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b77077-7740-44a1-88a3-cc5c83f0cce7-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:37.916324 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:37.916194    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbll\" (UniqueName: \"kubernetes.io/projected/91b77077-7740-44a1-88a3-cc5c83f0cce7-kube-api-access-vtbll\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:03:38.487050 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.486998    2577 generic.go:356] "Generic (PLEG): container finished" podID="91b77077-7740-44a1-88a3-cc5c83f0cce7" containerID="604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2" exitCode=0
Jul 29 02:03:38.487253 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.487063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"
Jul 29 02:03:38.487253 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.487081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" event={"ID":"91b77077-7740-44a1-88a3-cc5c83f0cce7","Type":"ContainerDied","Data":"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"}
Jul 29 02:03:38.487253 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.487121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq" event={"ID":"91b77077-7740-44a1-88a3-cc5c83f0cce7","Type":"ContainerDied","Data":"e674c94ac2b1c1c696ed53e8dfa876ac7c2d76ecdab09672052dd76bd800558b"}
Jul 29 02:03:38.487253 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.487137    2577 scope.go:117] "RemoveContainer" containerID="604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"
Jul 29 02:03:38.495768 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.495755    2577 scope.go:117] "RemoveContainer" containerID="604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"
Jul 29 02:03:38.496056 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:03:38.496020    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2\": container with ID starting with 604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2 not found: ID does not exist" containerID="604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"
Jul 29 02:03:38.496107 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.496083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2"} err="failed to get container status \"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2\": rpc error: code = NotFound desc = could not find container \"604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2\": container with ID starting with 604070e336a6e282734f2c4d4c626c0068a3abac492ffe753dca823c437a70c2 not found: ID does not exist"
Jul 29 02:03:38.507935 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.507911    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:03:38.514431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:38.514413    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/sub-test-model-a7bf0d-kserve-84959fd547-7mbvq"]
Jul 29 02:03:39.466582 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:39.466551    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" path="/var/lib/kubelet/pods/88723e4d-7023-42c6-9dd3-8bf50cfcecd2/volumes"
Jul 29 02:03:39.466946 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:39.466924    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b77077-7740-44a1-88a3-cc5c83f0cce7" path="/var/lib/kubelet/pods/91b77077-7740-44a1-88a3-cc5c83f0cce7/volumes"
Jul 29 02:03:42.502353 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:42.502323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:42.502786 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:42.502449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:42.515594 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:42.515570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:03:43.523287 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:03:43.523253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:04:03.960422 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960388    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960734    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" containerName="main"
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960745    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" containerName="main"
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960768    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b77077-7740-44a1-88a3-cc5c83f0cce7" containerName="main"
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960774    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b77077-7740-44a1-88a3-cc5c83f0cce7" containerName="main"
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960830    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b77077-7740-44a1-88a3-cc5c83f0cce7" containerName="main"
Jul 29 02:04:03.961265 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.960841    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="88723e4d-7023-42c6-9dd3-8bf50cfcecd2" containerName="main"
Jul 29 02:04:03.963856 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.963839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:03.966500 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.966480    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"kube-root-ca.crt\""
Jul 29 02:04:03.967605 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.967588    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"openshift-service-ca.crt\""
Jul 29 02:04:03.967759 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.967723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"default-dockercfg-78mbk\""
Jul 29 02:04:03.967870 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.967759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7bf0d\"/\"rate-test-model-a7bf0d-kserve-self-signed-certs\""
Jul 29 02:04:03.972585 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:03.972565    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:04:04.039431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.039556 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.039556 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.039556 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.039706 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl94\" (UniqueName: \"kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.039706 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.039598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140287 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140406 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140406 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140500 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl94\" (UniqueName: \"kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140500 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140584 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140759 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140759 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.140914 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.140801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.142361 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.142340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.143013 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.142992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.150165 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.150144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl94\" (UniqueName: \"kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94\") pod \"rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") " pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.273511 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.273438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:04.403189 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.403164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:04:04.405073 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:04:04.405033    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b45651_ddfc_4a4c_8388_8ebbcf60f786.slice/crio-7b1e49077b4cbbc9286146abee9e48b5d1af651b2c882a5142468dafffd9c4ff WatchSource:0}: Error finding container 7b1e49077b4cbbc9286146abee9e48b5d1af651b2c882a5142468dafffd9c4ff: Status 404 returned error can't find the container with id 7b1e49077b4cbbc9286146abee9e48b5d1af651b2c882a5142468dafffd9c4ff
Jul 29 02:04:04.611978 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.611908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" event={"ID":"e7b45651-ddfc-4a4c-8388-8ebbcf60f786","Type":"ContainerStarted","Data":"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"}
Jul 29 02:04:04.611978 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.611952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" event={"ID":"e7b45651-ddfc-4a4c-8388-8ebbcf60f786","Type":"ContainerStarted","Data":"7b1e49077b4cbbc9286146abee9e48b5d1af651b2c882a5142468dafffd9c4ff"}
Jul 29 02:04:04.629871 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:04.629821    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" podStartSLOduration=1.629806496 podStartE2EDuration="1.629806496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:04:04.628364712 +0000 UTC m=+3231.781810089" watchObservedRunningTime="2026-07-29 02:04:04.629806496 +0000 UTC m=+3231.783251924"
Jul 29 02:04:07.256260 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:07.256228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:04:14.274586 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:14.274549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:14.275091 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:14.274600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:14.287172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:14.287148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:14.670719 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:14.670691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:04:40.567863 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:40.567829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:04:55.869162 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:04:55.869127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:05:10.166870 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:10.166833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:05:14.144096 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:14.144060    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:05:15.081555 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:15.081525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:05:27.049593 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:27.049556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:05:29.162337 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.162296    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" podUID="2c96dc16-1784-4d8d-8e93-d428846f08af" containerName="main" containerID="cri-o://90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858" gracePeriod=45
Jul 29 02:05:29.414406 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.414351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:05:29.576954 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.576921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh76j\" (UniqueName: \"kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577158 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.576972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577158 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577158 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577158 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577380 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm\") pod \"2c96dc16-1784-4d8d-8e93-d428846f08af\" (UID: \"2c96dc16-1784-4d8d-8e93-d428846f08af\") "
Jul 29 02:05:29.577380 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:29.577380 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:29.577380 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home" (OuterVolumeSpecName: "home") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:29.577605 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577529    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:29.577605 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577548    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:29.577605 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.577562    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:29.579261 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.579230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j" (OuterVolumeSpecName: "kube-api-access-qh76j") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "kube-api-access-qh76j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:05:29.579632 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.579613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm" (OuterVolumeSpecName: "dshm") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:29.579693 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.579639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c96dc16-1784-4d8d-8e93-d428846f08af" (UID: "2c96dc16-1784-4d8d-8e93-d428846f08af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:05:29.678476 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.678399    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh76j\" (UniqueName: \"kubernetes.io/projected/2c96dc16-1784-4d8d-8e93-d428846f08af-kube-api-access-qh76j\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:29.678476 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.678427    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c96dc16-1784-4d8d-8e93-d428846f08af-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:29.678476 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:29.678436    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c96dc16-1784-4d8d-8e93-d428846f08af-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.011550 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.011456    2577 generic.go:356] "Generic (PLEG): container finished" podID="2c96dc16-1784-4d8d-8e93-d428846f08af" containerID="90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858" exitCode=0
Jul 29 02:05:30.011550 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.011518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"
Jul 29 02:05:30.011780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.011541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" event={"ID":"2c96dc16-1784-4d8d-8e93-d428846f08af","Type":"ContainerDied","Data":"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"}
Jul 29 02:05:30.011780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.011593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm" event={"ID":"2c96dc16-1784-4d8d-8e93-d428846f08af","Type":"ContainerDied","Data":"be5d7550a383952da5433a9a93cdd0ee4d7fba0a591223b4026ebc12687b1c0c"}
Jul 29 02:05:30.011780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.011613    2577 scope.go:117] "RemoveContainer" containerID="90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"
Jul 29 02:05:30.020415 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.020399    2577 scope.go:117] "RemoveContainer" containerID="90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"
Jul 29 02:05:30.020674 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:05:30.020658    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858\": container with ID starting with 90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858 not found: ID does not exist" containerID="90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"
Jul 29 02:05:30.020716 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.020682    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858"} err="failed to get container status \"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858\": rpc error: code = NotFound desc = could not find container \"90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858\": container with ID starting with 90ef01c3ab6570da882a2cb7134da80b86dfcd2c3ec5614b39918023a7254858 not found: ID does not exist"
Jul 29 02:05:30.032478 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.032457    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:05:30.035733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.035710    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-27cc80/rate-test-model-27cc80-kserve-74f76b6f4c-sgxgm"]
Jul 29 02:05:30.100154 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.100099    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" podUID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" containerName="main" containerID="cri-o://4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4" gracePeriod=45
Jul 29 02:05:30.353662 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.353637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:05:30.485117 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485296 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485296 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggl94\" (UniqueName: \"kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485296 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485296 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485296 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs\") pod \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\" (UID: \"e7b45651-ddfc-4a4c-8388-8ebbcf60f786\") "
Jul 29 02:05:30.485553 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:30.485553 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home" (OuterVolumeSpecName: "home") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:30.485553 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:30.485682 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485638    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.485682 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485653    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.485682 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.485661    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.487307 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.487280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94" (OuterVolumeSpecName: "kube-api-access-ggl94") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "kube-api-access-ggl94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:05:30.487421 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.487330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm" (OuterVolumeSpecName: "dshm") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:05:30.487421 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.487363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7b45651-ddfc-4a4c-8388-8ebbcf60f786" (UID: "e7b45651-ddfc-4a4c-8388-8ebbcf60f786"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:05:30.586830 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.586808    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.586830 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.586827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggl94\" (UniqueName: \"kubernetes.io/projected/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-kube-api-access-ggl94\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:30.586987 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:30.586838    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b45651-ddfc-4a4c-8388-8ebbcf60f786-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:05:31.018134 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.018031    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" containerID="4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4" exitCode=0
Jul 29 02:05:31.018134 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.018073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" event={"ID":"e7b45651-ddfc-4a4c-8388-8ebbcf60f786","Type":"ContainerDied","Data":"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"}
Jul 29 02:05:31.018134 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.018092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"
Jul 29 02:05:31.018134 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.018123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s" event={"ID":"e7b45651-ddfc-4a4c-8388-8ebbcf60f786","Type":"ContainerDied","Data":"7b1e49077b4cbbc9286146abee9e48b5d1af651b2c882a5142468dafffd9c4ff"}
Jul 29 02:05:31.018443 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.018141    2577 scope.go:117] "RemoveContainer" containerID="4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"
Jul 29 02:05:31.027341 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.027324    2577 scope.go:117] "RemoveContainer" containerID="4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"
Jul 29 02:05:31.027594 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:05:31.027576    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4\": container with ID starting with 4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4 not found: ID does not exist" containerID="4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"
Jul 29 02:05:31.027639 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.027601    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4"} err="failed to get container status \"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4\": rpc error: code = NotFound desc = could not find container \"4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4\": container with ID starting with 4510a0652da9e8e6e1d39f89e51f9c7b7b7cbf3ac5465f7764deba5b9f6423a4 not found: ID does not exist"
Jul 29 02:05:31.041196 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.041173    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:05:31.046171 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.046146    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7bf0d/rate-test-model-a7bf0d-kserve-7957d45f9f-fvb2s"]
Jul 29 02:05:31.466419 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.466382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c96dc16-1784-4d8d-8e93-d428846f08af" path="/var/lib/kubelet/pods/2c96dc16-1784-4d8d-8e93-d428846f08af/volumes"
Jul 29 02:05:31.466944 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:31.466923    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" path="/var/lib/kubelet/pods/e7b45651-ddfc-4a4c-8388-8ebbcf60f786/volumes"
Jul 29 02:05:47.466624 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:05:47.466593    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:06:07.876678 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.876638    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:06:07.877178 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" containerName="main"
Jul 29 02:06:07.877234 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877180    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" containerName="main"
Jul 29 02:06:07.877234 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c96dc16-1784-4d8d-8e93-d428846f08af" containerName="main"
Jul 29 02:06:07.877234 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877202    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c96dc16-1784-4d8d-8e93-d428846f08af" containerName="main"
Jul 29 02:06:07.877329 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877264    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c96dc16-1784-4d8d-8e93-d428846f08af" containerName="main"
Jul 29 02:06:07.877329 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.877273    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b45651-ddfc-4a4c-8388-8ebbcf60f786" containerName="main"
Jul 29 02:06:07.880276 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.880262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:07.882798 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.882773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e02970\""
Jul 29 02:06:07.891918 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.891897    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:06:07.896032 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.896013    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:06:07.899204 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.899188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:07.903256 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.903231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e02970-dockercfg-jhkgz\""
Jul 29 02:06:07.913292 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.913266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:06:07.977756 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.977730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:07.977877 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.977769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:07.977877 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.977794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7fw\" (UniqueName: \"kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:07.977877 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:07.977827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcm2\" (UniqueName: \"kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.079236 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.079431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:08.079431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7fw\" (UniqueName: \"kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:08.079431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcm2\" (UniqueName: \"kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.079968 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.080032 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.079950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:08.091961 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.091939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7fw\" (UniqueName: \"kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw\") pod \"payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:08.092320 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.092297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcm2\" (UniqueName: \"kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2\") pod \"payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.190404 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.190333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:08.208596 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.208568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:08.537792 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.537769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:06:08.540186 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:06:08.540158    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6cc545_4fe1_443a_88bc_a1400234a1ff.slice/crio-5a2bd7deb7bb237be755c082ae03cb8f9847cc96211f7813dc73d891d29fe2ed WatchSource:0}: Error finding container 5a2bd7deb7bb237be755c082ae03cb8f9847cc96211f7813dc73d891d29fe2ed: Status 404 returned error can't find the container with id 5a2bd7deb7bb237be755c082ae03cb8f9847cc96211f7813dc73d891d29fe2ed
Jul 29 02:06:08.565761 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:08.565741    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:06:08.574058 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:06:08.574010    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2383a1_6c96_4d98_a8ff_0e6c38d65c5c.slice/crio-53fc0466c28803f78f8e5b910237bd1c5213e42855bed4755116146aee6bfd69 WatchSource:0}: Error finding container 53fc0466c28803f78f8e5b910237bd1c5213e42855bed4755116146aee6bfd69: Status 404 returned error can't find the container with id 53fc0466c28803f78f8e5b910237bd1c5213e42855bed4755116146aee6bfd69
Jul 29 02:06:09.196803 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.196749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" event={"ID":"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c","Type":"ContainerStarted","Data":"ed1e2f40801881d16391f7026d4a54e800f30cef838463bdff9e2922fbe4d122"}
Jul 29 02:06:09.197293 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.196861    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:09.197293 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.196884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" event={"ID":"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c","Type":"ContainerStarted","Data":"53fc0466c28803f78f8e5b910237bd1c5213e42855bed4755116146aee6bfd69"}
Jul 29 02:06:09.198272 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.198247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" event={"ID":"ec6cc545-4fe1-443a-88bc-a1400234a1ff","Type":"ContainerStarted","Data":"2b6bf5774f674646c3ce54154f9a44e89e8997c8d1616a7c3bde5d7ecf52246a"}
Jul 29 02:06:09.198406 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.198297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:09.198406 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.198316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" event={"ID":"ec6cc545-4fe1-443a-88bc-a1400234a1ff","Type":"ContainerStarted","Data":"5a2bd7deb7bb237be755c082ae03cb8f9847cc96211f7813dc73d891d29fe2ed"}
Jul 29 02:06:09.213811 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.213770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" podStartSLOduration=2.2137584 podStartE2EDuration="2.2137584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:06:09.212645741 +0000 UTC m=+3356.366091118" watchObservedRunningTime="2026-07-29 02:06:09.2137584 +0000 UTC m=+3356.367203776"
Jul 29 02:06:09.229337 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:09.229294    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" podStartSLOduration=2.229281136 podStartE2EDuration="2.229281136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:06:09.227647944 +0000 UTC m=+3356.381093320" watchObservedRunningTime="2026-07-29 02:06:09.229281136 +0000 UTC m=+3356.382726511"
Jul 29 02:06:16.266740 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:16.266702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:06:20.205599 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:20.205564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:06:20.206089 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:20.205726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:06:38.819164 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:38.819136    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v_ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c/payload-processing/0.log"
Jul 29 02:06:50.267348 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.267316    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:06:50.270829 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.270813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.273826 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.273804    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-74cb2e\"/\"openshift-service-ca.crt\""
Jul 29 02:06:50.274733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.274715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-74cb2e\"/\"default-dockercfg-xjd59\""
Jul 29 02:06:50.274932 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.274913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-74cb2e\"/\"kube-root-ca.crt\""
Jul 29 02:06:50.274996 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.274954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-74cb2e\"/\"ipp-route-74cb2e-kserve-self-signed-certs\""
Jul 29 02:06:50.280498 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.280479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:06:50.432526 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.432671 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.432671 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.432671 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rszx\" (UniqueName: \"kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.432797 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.432797 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.432736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533442 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533442 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533621 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533621 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533621 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rszx\" (UniqueName: \"kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533621 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533826 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533943 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.533943 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.533927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.535876 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.535855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.536130 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.536114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.542596 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.542573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rszx\" (UniqueName: \"kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx\") pod \"ipp-route-74cb2e-kserve-74bf46bf4c-c99zm\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") " pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.581243 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.581218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:06:50.710489 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:50.710466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:06:50.712818 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:06:50.712790    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea339ea_3ff1_4f5a_b0f8_500dd0b32727.slice/crio-7509e580909a3d6f94a0284e2ea47a3432e95daa1c34f2fde0972b76a4d6df24 WatchSource:0}: Error finding container 7509e580909a3d6f94a0284e2ea47a3432e95daa1c34f2fde0972b76a4d6df24: Status 404 returned error can't find the container with id 7509e580909a3d6f94a0284e2ea47a3432e95daa1c34f2fde0972b76a4d6df24
Jul 29 02:06:51.394764 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:51.394732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" event={"ID":"dea339ea-3ff1-4f5a-b0f8-500dd0b32727","Type":"ContainerStarted","Data":"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"}
Jul 29 02:06:51.395228 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:51.394781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" event={"ID":"dea339ea-3ff1-4f5a-b0f8-500dd0b32727","Type":"ContainerStarted","Data":"7509e580909a3d6f94a0284e2ea47a3432e95daa1c34f2fde0972b76a4d6df24"}
Jul 29 02:06:51.415797 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:51.415747    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" podStartSLOduration=1.41573322 podStartE2EDuration="1.41573322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:06:51.413997356 +0000 UTC m=+3398.567442732" watchObservedRunningTime="2026-07-29 02:06:51.41573322 +0000 UTC m=+3398.569178596"
Jul 29 02:06:53.964477 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:06:53.964442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:07:00.582112 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:00.582077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:07:00.582491 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:00.582125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:07:00.594319 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:00.594285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:07:01.452777 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:01.452744    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:07:24.552530 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:24.552452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:07:36.184914 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:36.184882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v_ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c/payload-processing/0.log"
Jul 29 02:07:40.355181 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.355146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:07:40.358507 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.358491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.361286 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.361263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5be2ca\""
Jul 29 02:07:40.361701 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.361685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5be2ca-dockercfg-rb555\""
Jul 29 02:07:40.371471 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.371447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:07:40.464723 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.464680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzzb\" (UniqueName: \"kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.464907 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.464751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.565704 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.565676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.565851 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.565782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzzb\" (UniqueName: \"kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.566432 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.566413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.575336 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.575311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzzb\" (UniqueName: \"kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb\") pod \"payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:40.668390 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:40.668303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:41.008805 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:41.008778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:07:41.010772 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:07:41.010745    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b83dbd_4964_4780_abba_69fd21de1205.slice/crio-8eba399db8a090fa002ad116adbf10af006a8cd91a72f6c4da3aa11a0a733b5b WatchSource:0}: Error finding container 8eba399db8a090fa002ad116adbf10af006a8cd91a72f6c4da3aa11a0a733b5b: Status 404 returned error can't find the container with id 8eba399db8a090fa002ad116adbf10af006a8cd91a72f6c4da3aa11a0a733b5b
Jul 29 02:07:41.639766 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:41.639729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" event={"ID":"c1b83dbd-4964-4780-abba-69fd21de1205","Type":"ContainerStarted","Data":"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"}
Jul 29 02:07:41.640261 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:41.639780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" event={"ID":"c1b83dbd-4964-4780-abba-69fd21de1205","Type":"ContainerStarted","Data":"8eba399db8a090fa002ad116adbf10af006a8cd91a72f6c4da3aa11a0a733b5b"}
Jul 29 02:07:41.640261 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:41.639864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:07:41.660843 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:41.660799    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" podStartSLOduration=1.6607847 podStartE2EDuration="1.6607847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:07:41.658978026 +0000 UTC m=+3448.812423399" watchObservedRunningTime="2026-07-29 02:07:41.6607847 +0000 UTC m=+3448.814230076"
Jul 29 02:07:49.154665 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:49.154630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:07:52.647211 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:07:52.647181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:08:05.007403 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.007368    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"]
Jul 29 02:08:05.010688 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.010671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:05.013480 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.013455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 02:08:05.013600 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.013559    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 02:08:05.019813 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.019788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"]
Jul 29 02:08:05.068648 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.068620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg\" (UID: \"02a95957-43ad-4c4e-b82e-07f6ae2c858b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:05.169290 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.169265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg\" (UID: \"02a95957-43ad-4c4e-b82e-07f6ae2c858b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:05.169887 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.169866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg\" (UID: \"02a95957-43ad-4c4e-b82e-07f6ae2c858b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:05.322866 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.322842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:05.655006 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.654903    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"]
Jul 29 02:08:05.657828 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:08:05.657803    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a95957_43ad_4c4e_b82e_07f6ae2c858b.slice/crio-17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965 WatchSource:0}: Error finding container 17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965: Status 404 returned error can't find the container with id 17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965
Jul 29 02:08:05.760929 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:05.760900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg" event={"ID":"02a95957-43ad-4c4e-b82e-07f6ae2c858b","Type":"ContainerStarted","Data":"17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965"}
Jul 29 02:08:06.766695 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:06.766659    2577 generic.go:356] "Generic (PLEG): container finished" podID="02a95957-43ad-4c4e-b82e-07f6ae2c858b" containerID="eb1a207e0265471069a22fa00df22802ec4fea337e1be4bd02846c6f0183e135" exitCode=0
Jul 29 02:08:06.767074 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:06.766742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg" event={"ID":"02a95957-43ad-4c4e-b82e-07f6ae2c858b","Type":"ContainerDied","Data":"eb1a207e0265471069a22fa00df22802ec4fea337e1be4bd02846c6f0183e135"}
Jul 29 02:08:07.902407 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:07.902377    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:07.992778 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:07.992742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca\") pod \"02a95957-43ad-4c4e-b82e-07f6ae2c858b\" (UID: \"02a95957-43ad-4c4e-b82e-07f6ae2c858b\") "
Jul 29 02:08:07.993140 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:07.993117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02a95957-43ad-4c4e-b82e-07f6ae2c858b" (UID: "02a95957-43ad-4c4e-b82e-07f6ae2c858b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:08:08.093574 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:08.093541    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a95957-43ad-4c4e-b82e-07f6ae2c858b-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:08.776576 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:08.776541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg" event={"ID":"02a95957-43ad-4c4e-b82e-07f6ae2c858b","Type":"ContainerDied","Data":"17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965"}
Jul 29 02:08:08.776576 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:08.776581    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"
Jul 29 02:08:08.776789 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:08.776591    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17252f90995166cffdfcc4275716b513515f0bc20565180288534f27525df965"
Jul 29 02:08:10.046586 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.046548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:08:10.047067 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.046818    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" podUID="c1b83dbd-4964-4780-abba-69fd21de1205" containerName="payload-processing" containerID="cri-o://8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57" gracePeriod=30
Jul 29 02:08:10.189709 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.189674    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"]
Jul 29 02:08:10.191422 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.191398    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5be2ca-rx2gg"]
Jul 29 02:08:10.314706 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.314683    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:08:10.414916 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.414886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkzzb\" (UniqueName: \"kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb\") pod \"c1b83dbd-4964-4780-abba-69fd21de1205\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") "
Jul 29 02:08:10.415113 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.414995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume\") pod \"c1b83dbd-4964-4780-abba-69fd21de1205\" (UID: \"c1b83dbd-4964-4780-abba-69fd21de1205\") "
Jul 29 02:08:10.415366 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.415342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1b83dbd-4964-4780-abba-69fd21de1205" (UID: "c1b83dbd-4964-4780-abba-69fd21de1205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:08:10.416925 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.416905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb" (OuterVolumeSpecName: "kube-api-access-dkzzb") pod "c1b83dbd-4964-4780-abba-69fd21de1205" (UID: "c1b83dbd-4964-4780-abba-69fd21de1205"). InnerVolumeSpecName "kube-api-access-dkzzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:08:10.516146 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.516104    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b83dbd-4964-4780-abba-69fd21de1205-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:10.516146 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.516135    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzzb\" (UniqueName: \"kubernetes.io/projected/c1b83dbd-4964-4780-abba-69fd21de1205-kube-api-access-dkzzb\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:10.786450 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.786417    2577 generic.go:356] "Generic (PLEG): container finished" podID="c1b83dbd-4964-4780-abba-69fd21de1205" containerID="8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57" exitCode=0
Jul 29 02:08:10.786615 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.786456    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"
Jul 29 02:08:10.786615 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.786503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" event={"ID":"c1b83dbd-4964-4780-abba-69fd21de1205","Type":"ContainerDied","Data":"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"}
Jul 29 02:08:10.786615 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.786557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d" event={"ID":"c1b83dbd-4964-4780-abba-69fd21de1205","Type":"ContainerDied","Data":"8eba399db8a090fa002ad116adbf10af006a8cd91a72f6c4da3aa11a0a733b5b"}
Jul 29 02:08:10.786615 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.786584    2577 scope.go:117] "RemoveContainer" containerID="8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"
Jul 29 02:08:10.795283 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.795261    2577 scope.go:117] "RemoveContainer" containerID="8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"
Jul 29 02:08:10.795539 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:08:10.795523    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57\": container with ID starting with 8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57 not found: ID does not exist" containerID="8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"
Jul 29 02:08:10.795592 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.795546    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57"} err="failed to get container status \"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57\": rpc error: code = NotFound desc = could not find container \"8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57\": container with ID starting with 8f6e4eade697c5eb909f5e6100f2a07b884370342a20b16057ab1f9f99035e57 not found: ID does not exist"
Jul 29 02:08:10.807431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.807408    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:08:10.810300 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:10.810280    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5be2ca-77d9bbf86d-5w28d"]
Jul 29 02:08:11.468036 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:11.468000    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a95957-43ad-4c4e-b82e-07f6ae2c858b" path="/var/lib/kubelet/pods/02a95957-43ad-4c4e-b82e-07f6ae2c858b/volumes"
Jul 29 02:08:11.468431 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:11.468415    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b83dbd-4964-4780-abba-69fd21de1205" path="/var/lib/kubelet/pods/c1b83dbd-4964-4780-abba-69fd21de1205/volumes"
Jul 29 02:08:15.247292 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:15.247247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:08:35.050473 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:35.050390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:08:41.631243 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:41.631204    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:08:45.747466 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:45.747432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:08:56.650648 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:56.650608    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" podUID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" containerName="main" containerID="cri-o://adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3" gracePeriod=45
Jul 29 02:08:56.895851 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:56.895826    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:08:57.006862 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.006772    2577 generic.go:356] "Generic (PLEG): container finished" podID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" containerID="adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3" exitCode=0
Jul 29 02:08:57.006862 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.006821    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"
Jul 29 02:08:57.006862 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.006835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" event={"ID":"dea339ea-3ff1-4f5a-b0f8-500dd0b32727","Type":"ContainerDied","Data":"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"}
Jul 29 02:08:57.007097 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.006877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm" event={"ID":"dea339ea-3ff1-4f5a-b0f8-500dd0b32727","Type":"ContainerDied","Data":"7509e580909a3d6f94a0284e2ea47a3432e95daa1c34f2fde0972b76a4d6df24"}
Jul 29 02:08:57.007097 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.006901    2577 scope.go:117] "RemoveContainer" containerID="adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"
Jul 29 02:08:57.011195 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rszx\" (UniqueName: \"kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011558 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm\") pod \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\" (UID: \"dea339ea-3ff1-4f5a-b0f8-500dd0b32727\") "
Jul 29 02:08:57.011558 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache" (OuterVolumeSpecName: "model-cache") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:08:57.011667 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home" (OuterVolumeSpecName: "home") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:08:57.011667 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:08:57.011788 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011685    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.011788 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011703    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.011788 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.011717    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.013459 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.013425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 02:08:57.013549 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.013523    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm" (OuterVolumeSpecName: "dshm") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:08:57.013598 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.013541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx" (OuterVolumeSpecName: "kube-api-access-5rszx") pod "dea339ea-3ff1-4f5a-b0f8-500dd0b32727" (UID: "dea339ea-3ff1-4f5a-b0f8-500dd0b32727"). InnerVolumeSpecName "kube-api-access-5rszx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:08:57.016238 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.016223    2577 scope.go:117] "RemoveContainer" containerID="adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"
Jul 29 02:08:57.016491 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:08:57.016470    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3\": container with ID starting with adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3 not found: ID does not exist" containerID="adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"
Jul 29 02:08:57.016548 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.016504    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3"} err="failed to get container status \"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3\": rpc error: code = NotFound desc = could not find container \"adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3\": container with ID starting with adc21dc59ca4e3416816ee4790c3847bdfc2eac934a27192d9279259fd17cbc3 not found: ID does not exist"
Jul 29 02:08:57.112266 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.112241    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.112266 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.112263    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rszx\" (UniqueName: \"kubernetes.io/projected/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-kube-api-access-5rszx\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.112266 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.112272    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea339ea-3ff1-4f5a-b0f8-500dd0b32727-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:08:57.328711 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.328675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:08:57.333243 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.333220    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-74cb2e/ipp-route-74cb2e-kserve-74bf46bf4c-c99zm"]
Jul 29 02:08:57.466991 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:08:57.466955    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" path="/var/lib/kubelet/pods/dea339ea-3ff1-4f5a-b0f8-500dd0b32727/volumes"
Jul 29 02:09:46.991927 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:46.991893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:09:46.992472 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:46.992191    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" podUID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" containerName="payload-processing" containerID="cri-o://ed1e2f40801881d16391f7026d4a54e800f30cef838463bdff9e2922fbe4d122" gracePeriod=30
Jul 29 02:09:47.002187 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.002160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:09:47.002363 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.002343    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" podUID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" containerName="payload-pre-processing" containerID="cri-o://2b6bf5774f674646c3ce54154f9a44e89e8997c8d1616a7c3bde5d7ecf52246a" gracePeriod=30
Jul 29 02:09:47.249336 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.249308    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" containerID="ed1e2f40801881d16391f7026d4a54e800f30cef838463bdff9e2922fbe4d122" exitCode=0
Jul 29 02:09:47.249477 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.249381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" event={"ID":"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c","Type":"ContainerDied","Data":"ed1e2f40801881d16391f7026d4a54e800f30cef838463bdff9e2922fbe4d122"}
Jul 29 02:09:47.251269 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.251246    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" containerID="2b6bf5774f674646c3ce54154f9a44e89e8997c8d1616a7c3bde5d7ecf52246a" exitCode=0
Jul 29 02:09:47.251380 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.251291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" event={"ID":"ec6cc545-4fe1-443a-88bc-a1400234a1ff","Type":"ContainerDied","Data":"2b6bf5774f674646c3ce54154f9a44e89e8997c8d1616a7c3bde5d7ecf52246a"}
Jul 29 02:09:47.276770 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.276751    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:09:47.280133 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.280117    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:09:47.328611 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.328582    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp7fw\" (UniqueName: \"kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw\") pod \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") "
Jul 29 02:09:47.328780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.328624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume\") pod \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") "
Jul 29 02:09:47.328780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.328661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume\") pod \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\" (UID: \"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c\") "
Jul 29 02:09:47.328780 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.328692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppcm2\" (UniqueName: \"kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2\") pod \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\" (UID: \"ec6cc545-4fe1-443a-88bc-a1400234a1ff\") "
Jul 29 02:09:47.329097 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.329065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec6cc545-4fe1-443a-88bc-a1400234a1ff" (UID: "ec6cc545-4fe1-443a-88bc-a1400234a1ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:09:47.329242 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.329214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" (UID: "ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:09:47.330856 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.330826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw" (OuterVolumeSpecName: "kube-api-access-rp7fw") pod "ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" (UID: "ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c"). InnerVolumeSpecName "kube-api-access-rp7fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:09:47.330995 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.330974    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2" (OuterVolumeSpecName: "kube-api-access-ppcm2") pod "ec6cc545-4fe1-443a-88bc-a1400234a1ff" (UID: "ec6cc545-4fe1-443a-88bc-a1400234a1ff"). InnerVolumeSpecName "kube-api-access-ppcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:09:47.429712 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.429683    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7fw\" (UniqueName: \"kubernetes.io/projected/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-kube-api-access-rp7fw\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:09:47.429712 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.429708    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6cc545-4fe1-443a-88bc-a1400234a1ff-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:09:47.429712 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.429717    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:09:47.429967 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:47.429726    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcm2\" (UniqueName: \"kubernetes.io/projected/ec6cc545-4fe1-443a-88bc-a1400234a1ff-kube-api-access-ppcm2\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:09:48.257155 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.257071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v" event={"ID":"ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c","Type":"ContainerDied","Data":"53fc0466c28803f78f8e5b910237bd1c5213e42855bed4755116146aee6bfd69"}
Jul 29 02:09:48.257155 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.257088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"
Jul 29 02:09:48.257628 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.257135    2577 scope.go:117] "RemoveContainer" containerID="ed1e2f40801881d16391f7026d4a54e800f30cef838463bdff9e2922fbe4d122"
Jul 29 02:09:48.258508 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.258427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw" event={"ID":"ec6cc545-4fe1-443a-88bc-a1400234a1ff","Type":"ContainerDied","Data":"5a2bd7deb7bb237be755c082ae03cb8f9847cc96211f7813dc73d891d29fe2ed"}
Jul 29 02:09:48.258508 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.258445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"
Jul 29 02:09:48.265858 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.265837    2577 scope.go:117] "RemoveContainer" containerID="2b6bf5774f674646c3ce54154f9a44e89e8997c8d1616a7c3bde5d7ecf52246a"
Jul 29 02:09:48.277346 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.277319    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:09:48.280098 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.280079    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e02970-6c957db8b6-dwz9v"]
Jul 29 02:09:48.305448 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.305426    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:09:48.309841 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:48.309819    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e02970-7bf5bb9c68-j4mnw"]
Jul 29 02:09:49.466755 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:49.466714    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" path="/var/lib/kubelet/pods/ec6cc545-4fe1-443a-88bc-a1400234a1ff/volumes"
Jul 29 02:09:49.467276 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:49.467255    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" path="/var/lib/kubelet/pods/ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c/volumes"
Jul 29 02:09:50.296683 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:50.296651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:09:54.460750 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.460715    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"]
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461073    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" containerName="payload-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461085    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" containerName="payload-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" containerName="payload-pre-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461106    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" containerName="payload-pre-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461113    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b83dbd-4964-4780-abba-69fd21de1205" containerName="payload-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461118    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b83dbd-4964-4780-abba-69fd21de1205" containerName="payload-processing"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" containerName="main"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461138    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" containerName="main"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461146    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a95957-43ad-4c4e-b82e-07f6ae2c858b" containerName="revoke-keys"
Jul 29 02:09:54.461172 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461152    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a95957-43ad-4c4e-b82e-07f6ae2c858b" containerName="revoke-keys"
Jul 29 02:09:54.461520 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461231    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b83dbd-4964-4780-abba-69fd21de1205" containerName="payload-processing"
Jul 29 02:09:54.461520 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461240    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6cc545-4fe1-443a-88bc-a1400234a1ff" containerName="payload-pre-processing"
Jul 29 02:09:54.461520 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461247    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a95957-43ad-4c4e-b82e-07f6ae2c858b" containerName="revoke-keys"
Jul 29 02:09:54.461520 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461255    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2383a1-6c96-4d98-a8ff-0e6c38d65c5c" containerName="payload-processing"
Jul 29 02:09:54.461520 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.461261    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea339ea-3ff1-4f5a-b0f8-500dd0b32727" containerName="main"
Jul 29 02:09:54.465930 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.465911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:54.469791 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.469763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b5c94\""
Jul 29 02:09:54.469910 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.469796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 02:09:54.474815 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.474786    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"]
Jul 29 02:09:54.487592 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.487479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k\" (UID: \"a489a4ce-13f7-4499-a0a1-d40946031f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:54.587869 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.587842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k\" (UID: \"a489a4ce-13f7-4499-a0a1-d40946031f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:54.588492 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.588461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k\" (UID: \"a489a4ce-13f7-4499-a0a1-d40946031f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:54.777620 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.777537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:54.907521 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:54.907499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"]
Jul 29 02:09:54.909689 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:09:54.909660    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489a4ce_13f7_4499_a0a1_d40946031f22.slice/crio-6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78 WatchSource:0}: Error finding container 6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78: Status 404 returned error can't find the container with id 6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78
Jul 29 02:09:55.293366 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:55.293335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k" event={"ID":"a489a4ce-13f7-4499-a0a1-d40946031f22","Type":"ContainerStarted","Data":"96c98f6829eac9518979cce92a48630ac9cf4cea45d9676a7e64bea2cc729153"}
Jul 29 02:09:55.293511 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:55.293398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k" event={"ID":"a489a4ce-13f7-4499-a0a1-d40946031f22","Type":"ContainerStarted","Data":"6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78"}
Jul 29 02:09:55.319574 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:55.319522    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k" podStartSLOduration=1.319508792 podStartE2EDuration="1.319508792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:09:55.317882473 +0000 UTC m=+3582.471327849" watchObservedRunningTime="2026-07-29 02:09:55.319508792 +0000 UTC m=+3582.472954168"
Jul 29 02:09:56.299166 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:56.299134    2577 generic.go:356] "Generic (PLEG): container finished" podID="a489a4ce-13f7-4499-a0a1-d40946031f22" containerID="96c98f6829eac9518979cce92a48630ac9cf4cea45d9676a7e64bea2cc729153" exitCode=0
Jul 29 02:09:56.299565 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:56.299199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k" event={"ID":"a489a4ce-13f7-4499-a0a1-d40946031f22","Type":"ContainerDied","Data":"96c98f6829eac9518979cce92a48630ac9cf4cea45d9676a7e64bea2cc729153"}
Jul 29 02:09:57.431621 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:57.431589    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:57.515797 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:57.515770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca\") pod \"a489a4ce-13f7-4499-a0a1-d40946031f22\" (UID: \"a489a4ce-13f7-4499-a0a1-d40946031f22\") "
Jul 29 02:09:57.516103 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:57.516080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a489a4ce-13f7-4499-a0a1-d40946031f22" (UID: "a489a4ce-13f7-4499-a0a1-d40946031f22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:09:57.616826 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:57.616808    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a489a4ce-13f7-4499-a0a1-d40946031f22-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:09:58.309375 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:58.309344    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"
Jul 29 02:09:58.309375 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:58.309362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k" event={"ID":"a489a4ce-13f7-4499-a0a1-d40946031f22","Type":"ContainerDied","Data":"6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78"}
Jul 29 02:09:58.309557 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:09:58.309431    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6389f148dab54aaa717b31bf1de8fa4f8a173a758a4f66242e6f61637f861b78"
Jul 29 02:10:05.114593 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.114511    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 02:10:05.115028 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.114788    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" podUID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" containerName="payload-processing" containerID="cri-o://cf702df7a0222f7efa8aa84d0ae71a5933ad3fbd7f7b0b4be318ff764b898de2" gracePeriod=30
Jul 29 02:10:05.122528 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.122503    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 02:10:05.278686 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.278656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"]
Jul 29 02:10:05.280168 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.280148    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-27cc80-xc96k"]
Jul 29 02:10:05.346250 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.346214    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" containerID="cf702df7a0222f7efa8aa84d0ae71a5933ad3fbd7f7b0b4be318ff764b898de2" exitCode=0
Jul 29 02:10:05.346324 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.346249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" event={"ID":"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5","Type":"ContainerDied","Data":"cf702df7a0222f7efa8aa84d0ae71a5933ad3fbd7f7b0b4be318ff764b898de2"}
Jul 29 02:10:05.346324 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.346317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z" event={"ID":"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5","Type":"ContainerDied","Data":"a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee"}
Jul 29 02:10:05.346429 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.346342    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63b13626c49792b33b9414e7c4180d55c46d290934cc7fd4b5add5b275094ee"
Jul 29 02:10:05.346545 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.346525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" podUID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" containerName="payload-pre-processing" containerID="cri-o://273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460" gracePeriod=30
Jul 29 02:10:05.407115 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.407066    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 02:10:05.466180 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.466150    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a489a4ce-13f7-4499-a0a1-d40946031f22" path="/var/lib/kubelet/pods/a489a4ce-13f7-4499-a0a1-d40946031f22/volumes"
Jul 29 02:10:05.483715 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.483688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume\") pod \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") "
Jul 29 02:10:05.483832 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.483752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skc7q\" (UniqueName: \"kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q\") pod \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\" (UID: \"7e30741f-4aed-4931-9a1f-cc02d6e2bfe5\") "
Jul 29 02:10:05.484138 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.484099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" (UID: "7e30741f-4aed-4931-9a1f-cc02d6e2bfe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:10:05.485895 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.485872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q" (OuterVolumeSpecName: "kube-api-access-skc7q") pod "7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" (UID: "7e30741f-4aed-4931-9a1f-cc02d6e2bfe5"). InnerVolumeSpecName "kube-api-access-skc7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:05.582865 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.582835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 02:10:05.584932 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.584905    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:05.584932 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.584927    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skc7q\" (UniqueName: \"kubernetes.io/projected/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5-kube-api-access-skc7q\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:05.685993 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.685928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume\") pod \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") "
Jul 29 02:10:05.686152 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.685998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbbcr\" (UniqueName: \"kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr\") pod \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\" (UID: \"4bee2796-45f3-46a3-af75-0f3a8bedc5ae\") "
Jul 29 02:10:05.686351 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.686326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bee2796-45f3-46a3-af75-0f3a8bedc5ae" (UID: "4bee2796-45f3-46a3-af75-0f3a8bedc5ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:10:05.687942 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.687924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr" (OuterVolumeSpecName: "kube-api-access-jbbcr") pod "4bee2796-45f3-46a3-af75-0f3a8bedc5ae" (UID: "4bee2796-45f3-46a3-af75-0f3a8bedc5ae"). InnerVolumeSpecName "kube-api-access-jbbcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:05.786742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.786716    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:05.786742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:05.786739    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbbcr\" (UniqueName: \"kubernetes.io/projected/4bee2796-45f3-46a3-af75-0f3a8bedc5ae-kube-api-access-jbbcr\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:06.352513 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352483    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" containerID="273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460" exitCode=0
Jul 29 02:10:06.352845 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352522    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"
Jul 29 02:10:06.352845 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" event={"ID":"4bee2796-45f3-46a3-af75-0f3a8bedc5ae","Type":"ContainerDied","Data":"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"}
Jul 29 02:10:06.352845 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr" event={"ID":"4bee2796-45f3-46a3-af75-0f3a8bedc5ae","Type":"ContainerDied","Data":"399be020511a36f41f4ab491ff7e9d761931f6f4a3f8e5f1776161c75f09bc17"}
Jul 29 02:10:06.352845 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352634    2577 scope.go:117] "RemoveContainer" containerID="273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"
Jul 29 02:10:06.352845 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.352728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"
Jul 29 02:10:06.362487 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.362470    2577 scope.go:117] "RemoveContainer" containerID="273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"
Jul 29 02:10:06.362733 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:10:06.362707    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460\": container with ID starting with 273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460 not found: ID does not exist" containerID="273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"
Jul 29 02:10:06.362848 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.362742    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460"} err="failed to get container status \"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460\": rpc error: code = NotFound desc = could not find container \"273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460\": container with ID starting with 273e2ede0fb00ea48c5f2bcd2e46692702781f28c5b0a3e393a971ae7d568460 not found: ID does not exist"
Jul 29 02:10:06.377851 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.377831    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 02:10:06.381466 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.381419    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-27cc80-75575b9d86-h6h4z"]
Jul 29 02:10:06.394122 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.394100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 02:10:06.403960 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:06.403938    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-27cc80-597959bb6d-cscvr"]
Jul 29 02:10:07.466169 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:07.466140    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" path="/var/lib/kubelet/pods/4bee2796-45f3-46a3-af75-0f3a8bedc5ae/volumes"
Jul 29 02:10:07.466533 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:07.466483    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" path="/var/lib/kubelet/pods/7e30741f-4aed-4931-9a1f-cc02d6e2bfe5/volumes"
Jul 29 02:10:08.358806 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:08.358773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:10:12.198555 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:12.198523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 02:10:12.199018 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:12.198715    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" containerID="cri-o://1ca3511b1292d712cfb25679317068c0d34fa141fcc1f99ecd2f5b3a951b6bef" gracePeriod=30
Jul 29 02:10:13.779985 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:13.779953    2577 scope.go:117] "RemoveContainer" containerID="cf702df7a0222f7efa8aa84d0ae71a5933ad3fbd7f7b0b4be318ff764b898de2"
Jul 29 02:10:15.538237 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:15.538195    2577 patch_prober.go:28] interesting pod/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 02:10:15.538612 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:15.538256    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 02:10:17.406671 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.406638    2577 generic.go:356] "Generic (PLEG): container finished" podID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerID="1ca3511b1292d712cfb25679317068c0d34fa141fcc1f99ecd2f5b3a951b6bef" exitCode=0
Jul 29 02:10:17.407008 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.406711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerDied","Data":"1ca3511b1292d712cfb25679317068c0d34fa141fcc1f99ecd2f5b3a951b6bef"}
Jul 29 02:10:17.407008 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.406823    2577 scope.go:117] "RemoveContainer" containerID="3e9825a946d62f13fdc1dde8f3b85ac9d17da8d2e7ca8d3b92fe599158a5ab07"
Jul 29 02:10:17.448204 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.448182    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 02:10:17.582319 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582429 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582429 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582429 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582413    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582566 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582442    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582566 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582566 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb49s\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582566 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:10:17.582742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy\") pod \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\" (UID: \"5773bece-b2cc-4b29-bad2-9de1db64cf6a\") "
Jul 29 02:10:17.582742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:10:17.582911 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582802    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:10:17.582911 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582852    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.582911 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:10:17.582911 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.582869    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.583674 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.583647    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:10:17.584613 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.584590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s" (OuterVolumeSpecName: "kube-api-access-mb49s") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "kube-api-access-mb49s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:17.584772 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.584749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:17.584861 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.584786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 02:10:17.584861 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.584790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5773bece-b2cc-4b29-bad2-9de1db64cf6a" (UID: "5773bece-b2cc-4b29-bad2-9de1db64cf6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 02:10:17.683499 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683474    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683499 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683496    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683625 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683505    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683625 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683513    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683625 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683520    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5773bece-b2cc-4b29-bad2-9de1db64cf6a-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683625 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683527    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5773bece-b2cc-4b29-bad2-9de1db64cf6a-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:17.683625 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:17.683535    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb49s\" (UniqueName: \"kubernetes.io/projected/5773bece-b2cc-4b29-bad2-9de1db64cf6a-kube-api-access-mb49s\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:18.130295 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.130261    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 02:10:18.130598 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.130456    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" podUID="51b2036c-0206-456d-a82e-0241db1f6e6c" containerName="payload-processing" containerID="cri-o://e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc" gracePeriod=30
Jul 29 02:10:18.138342 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.138314    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 02:10:18.138798 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.138529    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" podUID="58892a28-c854-4bf3-abda-4a580960591c" containerName="payload-pre-processing" containerID="cri-o://17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969" gracePeriod=30
Jul 29 02:10:18.404683 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.404655    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 02:10:18.407667 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.407635    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 02:10:18.413745 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.413723    2577 generic.go:356] "Generic (PLEG): container finished" podID="51b2036c-0206-456d-a82e-0241db1f6e6c" containerID="e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc" exitCode=0
Jul 29 02:10:18.413867 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.413753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" event={"ID":"51b2036c-0206-456d-a82e-0241db1f6e6c","Type":"ContainerDied","Data":"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"}
Jul 29 02:10:18.413867 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.413775    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"
Jul 29 02:10:18.413867 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.413806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf" event={"ID":"51b2036c-0206-456d-a82e-0241db1f6e6c","Type":"ContainerDied","Data":"6b3cae9bbb96d25d79aecc09e41b98cf4c51283a513d0a41abf83f074e525aaa"}
Jul 29 02:10:18.413867 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.413835    2577 scope.go:117] "RemoveContainer" containerID="e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"
Jul 29 02:10:18.415270 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.415250    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"
Jul 29 02:10:18.415437 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.415267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5" event={"ID":"5773bece-b2cc-4b29-bad2-9de1db64cf6a","Type":"ContainerDied","Data":"446301f8dd5e511fdc61b2e3219f402b0e67fe19cf4eb7468e2aa55aa248c21f"}
Jul 29 02:10:18.416532 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.416511    2577 generic.go:356] "Generic (PLEG): container finished" podID="58892a28-c854-4bf3-abda-4a580960591c" containerID="17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969" exitCode=0
Jul 29 02:10:18.416633 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.416537    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"
Jul 29 02:10:18.416633 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.416554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" event={"ID":"58892a28-c854-4bf3-abda-4a580960591c","Type":"ContainerDied","Data":"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"}
Jul 29 02:10:18.416633 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.416581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv" event={"ID":"58892a28-c854-4bf3-abda-4a580960591c","Type":"ContainerDied","Data":"e78be7011134194755a9a3dbff7faac867d3922cbba9e4e2e5e7ffa336b45d4d"}
Jul 29 02:10:18.422847 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.422830    2577 scope.go:117] "RemoveContainer" containerID="e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"
Jul 29 02:10:18.423124 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:10:18.423100    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc\": container with ID starting with e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc not found: ID does not exist" containerID="e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"
Jul 29 02:10:18.423241 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.423126    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc"} err="failed to get container status \"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc\": rpc error: code = NotFound desc = could not find container \"e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc\": container with ID starting with e3485714949e1081830307e183a4d3f7d0f85f5bd9263b2af7405b3040a403dc not found: ID does not exist"
Jul 29 02:10:18.423241 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.423142    2577 scope.go:117] "RemoveContainer" containerID="1ca3511b1292d712cfb25679317068c0d34fa141fcc1f99ecd2f5b3a951b6bef"
Jul 29 02:10:18.431868 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.431852    2577 scope.go:117] "RemoveContainer" containerID="17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"
Jul 29 02:10:18.439261 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.439242    2577 scope.go:117] "RemoveContainer" containerID="17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"
Jul 29 02:10:18.439511 ip-10-0-136-251 kubenswrapper[2577]: E0729 02:10:18.439493    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969\": container with ID starting with 17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969 not found: ID does not exist" containerID="17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"
Jul 29 02:10:18.439578 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.439520    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969"} err="failed to get container status \"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969\": rpc error: code = NotFound desc = could not find container \"17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969\": container with ID starting with 17aacfb567d08982fc66f80b4411ea0ee3c239a615cc9de14fb66a23e389d969 not found: ID does not exist"
Jul 29 02:10:18.446929 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.446904    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 02:10:18.451247 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.451229    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-27cc80-openshift-default-7db7c7d8f6-lfhd5"]
Jul 29 02:10:18.490368 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9mwv\" (UniqueName: \"kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv\") pod \"51b2036c-0206-456d-a82e-0241db1f6e6c\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") "
Jul 29 02:10:18.490477 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490392    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume\") pod \"51b2036c-0206-456d-a82e-0241db1f6e6c\" (UID: \"51b2036c-0206-456d-a82e-0241db1f6e6c\") "
Jul 29 02:10:18.490477 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume\") pod \"58892a28-c854-4bf3-abda-4a580960591c\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") "
Jul 29 02:10:18.490554 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58ppg\" (UniqueName: \"kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg\") pod \"58892a28-c854-4bf3-abda-4a580960591c\" (UID: \"58892a28-c854-4bf3-abda-4a580960591c\") "
Jul 29 02:10:18.490827 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51b2036c-0206-456d-a82e-0241db1f6e6c" (UID: "51b2036c-0206-456d-a82e-0241db1f6e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:10:18.490827 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.490806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58892a28-c854-4bf3-abda-4a580960591c" (UID: "58892a28-c854-4bf3-abda-4a580960591c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 02:10:18.492463 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.492440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv" (OuterVolumeSpecName: "kube-api-access-z9mwv") pod "51b2036c-0206-456d-a82e-0241db1f6e6c" (UID: "51b2036c-0206-456d-a82e-0241db1f6e6c"). InnerVolumeSpecName "kube-api-access-z9mwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:18.492587 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.492570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg" (OuterVolumeSpecName: "kube-api-access-58ppg") pod "58892a28-c854-4bf3-abda-4a580960591c" (UID: "58892a28-c854-4bf3-abda-4a580960591c"). InnerVolumeSpecName "kube-api-access-58ppg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 02:10:18.592207 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.592178    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b2036c-0206-456d-a82e-0241db1f6e6c-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:18.592207 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.592204    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58892a28-c854-4bf3-abda-4a580960591c-plugins-config-volume\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:18.592364 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.592218    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ppg\" (UniqueName: \"kubernetes.io/projected/58892a28-c854-4bf3-abda-4a580960591c-kube-api-access-58ppg\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:18.592364 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.592230    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mwv\" (UniqueName: \"kubernetes.io/projected/51b2036c-0206-456d-a82e-0241db1f6e6c-kube-api-access-z9mwv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\""
Jul 29 02:10:18.736478 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.736452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 02:10:18.744874 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.744851    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a7bf0d-65f59d9cc8-bcctf"]
Jul 29 02:10:18.760292 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.760266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 02:10:18.765806 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:18.765786    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a7bf0d-9b46f7d9b-5gttv"]
Jul 29 02:10:19.465850 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:19.465810    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b2036c-0206-456d-a82e-0241db1f6e6c" path="/var/lib/kubelet/pods/51b2036c-0206-456d-a82e-0241db1f6e6c/volumes"
Jul 29 02:10:19.466306 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:19.466290    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" path="/var/lib/kubelet/pods/5773bece-b2cc-4b29-bad2-9de1db64cf6a/volumes"
Jul 29 02:10:19.466742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:19.466724    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58892a28-c854-4bf3-abda-4a580960591c" path="/var/lib/kubelet/pods/58892a28-c854-4bf3-abda-4a580960591c/volumes"
Jul 29 02:10:21.362935 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:21.362905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpvrw"]
Jul 29 02:10:26.185698 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:26.185665    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d64f5d668-jmpml_1d47d37d-5a0d-4350-8c85-fb088e8bfb33/authorino/0.log"
Jul 29 02:10:32.089473 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:32.089440    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tr5wm_d900390a-d129-4426-8525-5bbc998014da/server/0.log"
Jul 29 02:10:32.580115 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:32.580083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d7cdd4c-l25bs_ff586751-2ef6-467b-9381-4b177ab766c3/copy-manifests/0.log"
Jul 29 02:10:32.588673 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:32.588651    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d7cdd4c-l25bs_ff586751-2ef6-467b-9381-4b177ab766c3/manager/0.log"
Jul 29 02:10:34.791244 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:34.791205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d64f5d668-jmpml_1d47d37d-5a0d-4350-8c85-fb088e8bfb33/authorino/0.log"
Jul 29 02:10:35.522752 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:35.522721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gpvrw_8e9a7a8b-37c4-41f7-b1f3-b595e75b7664/limitador/0.log"
Jul 29 02:10:36.359053 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:36.359017    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86657cdbcd-glh4b_7b46c7ae-7f5a-47ed-b546-84147b0f102d/kube-auth-proxy/0.log"
Jul 29 02:10:37.755775 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:37.755741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-ldclj_e38965c0-3f59-493d-b3ec-66ece70a5706/main/0.log"
Jul 29 02:10:43.052530 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.052496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hd2h/must-gather-pbx65"]
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053520    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053528    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b2036c-0206-456d-a82e-0241db1f6e6c" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053550    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b2036c-0206-456d-a82e-0241db1f6e6c" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053569    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58892a28-c854-4bf3-abda-4a580960591c" containerName="payload-pre-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053578    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58892a28-c854-4bf3-abda-4a580960591c" containerName="payload-pre-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053615    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" containerName="payload-pre-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053624    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" containerName="payload-pre-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053636    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053645    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053664    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a489a4ce-13f7-4499-a0a1-d40946031f22" containerName="revoke-keys"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053673    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a489a4ce-13f7-4499-a0a1-d40946031f22" containerName="revoke-keys"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053896    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e30741f-4aed-4931-9a1f-cc02d6e2bfe5" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053921    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a489a4ce-13f7-4499-a0a1-d40946031f22" containerName="revoke-keys"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053930    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5773bece-b2cc-4b29-bad2-9de1db64cf6a" containerName="istio-proxy"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053937    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bee2796-45f3-46a3-af75-0f3a8bedc5ae" containerName="payload-pre-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053954    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b2036c-0206-456d-a82e-0241db1f6e6c" containerName="payload-processing"
Jul 29 02:10:43.056213 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.053964    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58892a28-c854-4bf3-abda-4a580960591c" containerName="payload-pre-processing"
Jul 29 02:10:43.058356 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.058328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.061922 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.061893    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hd2h\"/\"openshift-service-ca.crt\""
Jul 29 02:10:43.062815 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.062798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hd2h\"/\"default-dockercfg-8smzm\""
Jul 29 02:10:43.062815 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.062808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hd2h\"/\"kube-root-ca.crt\""
Jul 29 02:10:43.068364 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.068342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hd2h/must-gather-pbx65"]
Jul 29 02:10:43.083652 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.083632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176bfced-ed43-419e-80d7-b4bb663c3232-must-gather-output\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.083751 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.083657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx7s\" (UniqueName: \"kubernetes.io/projected/176bfced-ed43-419e-80d7-b4bb663c3232-kube-api-access-jhx7s\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.185024 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.184993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx7s\" (UniqueName: \"kubernetes.io/projected/176bfced-ed43-419e-80d7-b4bb663c3232-kube-api-access-jhx7s\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.185188 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.185154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176bfced-ed43-419e-80d7-b4bb663c3232-must-gather-output\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.185432 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.185413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176bfced-ed43-419e-80d7-b4bb663c3232-must-gather-output\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.200841 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.200819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx7s\" (UniqueName: \"kubernetes.io/projected/176bfced-ed43-419e-80d7-b4bb663c3232-kube-api-access-jhx7s\") pod \"must-gather-pbx65\" (UID: \"176bfced-ed43-419e-80d7-b4bb663c3232\") " pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.367915 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.367886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hd2h/must-gather-pbx65"
Jul 29 02:10:43.514805 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.514776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hd2h/must-gather-pbx65"]
Jul 29 02:10:43.519706 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:10:43.519681    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176bfced_ed43_419e_80d7_b4bb663c3232.slice/crio-6be3901975228f4e6c5a5b2ccf106b4ed162e0af2d00a638221095cb212c5916 WatchSource:0}: Error finding container 6be3901975228f4e6c5a5b2ccf106b4ed162e0af2d00a638221095cb212c5916: Status 404 returned error can't find the container with id 6be3901975228f4e6c5a5b2ccf106b4ed162e0af2d00a638221095cb212c5916
Jul 29 02:10:43.521407 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.521392    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 02:10:43.528806 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:43.528781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hd2h/must-gather-pbx65" event={"ID":"176bfced-ed43-419e-80d7-b4bb663c3232","Type":"ContainerStarted","Data":"6be3901975228f4e6c5a5b2ccf106b4ed162e0af2d00a638221095cb212c5916"}
Jul 29 02:10:44.535871 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:44.535848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hd2h/must-gather-pbx65" event={"ID":"176bfced-ed43-419e-80d7-b4bb663c3232","Type":"ContainerStarted","Data":"15a088592e3db787770cfd2837bbd48d6ee0bfac6c096e20738bad94fd3055b3"}
Jul 29 02:10:45.543710 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:45.543679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hd2h/must-gather-pbx65" event={"ID":"176bfced-ed43-419e-80d7-b4bb663c3232","Type":"ContainerStarted","Data":"f59887e61e46b3d8f08135989b27ba49dc4e6293f49a56dd11b2fab4c04ebc62"}
Jul 29 02:10:45.566612 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:45.566561    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hd2h/must-gather-pbx65" podStartSLOduration=1.656793631 podStartE2EDuration="2.566543634s" totalImagesPullingTime="909.750003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:10:45.564600285 +0000 UTC m=+3632.718045661" watchObservedRunningTime="2026-07-29 02:10:45.566543634 +0000 UTC m=+3632.719989009"
Jul 29 02:10:46.165416 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:46.165388    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdhzb_d5adf746-cbd1-47d6-a225-636fdbc225c4/global-pull-secret-syncer/0.log"
Jul 29 02:10:46.283415 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:46.283381    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bj86g_fa339f5a-ecb3-4616-8e1e-02dba66601d8/konnectivity-agent/0.log"
Jul 29 02:10:46.383724 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:46.383699    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-251.ec2.internal_1ee83001d783d1446eea0c77a1a727ba/haproxy/0.log"
Jul 29 02:10:51.306062 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:51.305299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d64f5d668-jmpml_1d47d37d-5a0d-4350-8c85-fb088e8bfb33/authorino/0.log"
Jul 29 02:10:51.741177 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:51.741144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gpvrw_8e9a7a8b-37c4-41f7-b1f3-b595e75b7664/limitador/0.log"
Jul 29 02:10:53.479359 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.479328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/alertmanager/0.log"
Jul 29 02:10:53.523382 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.523331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/config-reloader/0.log"
Jul 29 02:10:53.586910 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.586874    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/kube-rbac-proxy-web/0.log"
Jul 29 02:10:53.636891 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.636862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/kube-rbac-proxy/0.log"
Jul 29 02:10:53.689995 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.689948    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/kube-rbac-proxy-metric/0.log"
Jul 29 02:10:53.751235 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.751152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/prom-label-proxy/0.log"
Jul 29 02:10:53.820220 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:53.820188    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b376f79-786c-4c1e-992e-bef99cae8dc1/init-config-reloader/0.log"
Jul 29 02:10:54.312003 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.311975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bf4bl_9f979293-3c07-4821-9763-0166f1e82ed0/node-exporter/0.log"
Jul 29 02:10:54.339221 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.339167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bf4bl_9f979293-3c07-4821-9763-0166f1e82ed0/kube-rbac-proxy/0.log"
Jul 29 02:10:54.364869 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.364834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bf4bl_9f979293-3c07-4821-9763-0166f1e82ed0/init-textfile/0.log"
Jul 29 02:10:54.490826 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.490797    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-t6r6z_ce35a5fd-22ff-4427-b846-01bbcaa8cd9e/kube-rbac-proxy-main/0.log"
Jul 29 02:10:54.516664 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.516616    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-t6r6z_ce35a5fd-22ff-4427-b846-01bbcaa8cd9e/kube-rbac-proxy-self/0.log"
Jul 29 02:10:54.545891 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.545862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-t6r6z_ce35a5fd-22ff-4427-b846-01bbcaa8cd9e/openshift-state-metrics/0.log"
Jul 29 02:10:54.910573 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.910537    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cff774c6-fqsf7_4ac4b421-15f1-458c-b4ee-571fb9ff9fbb/telemeter-client/0.log"
Jul 29 02:10:54.951450 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.951422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cff774c6-fqsf7_4ac4b421-15f1-458c-b4ee-571fb9ff9fbb/reload/0.log"
Jul 29 02:10:54.980950 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:54.980919    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cff774c6-fqsf7_4ac4b421-15f1-458c-b4ee-571fb9ff9fbb/kube-rbac-proxy/0.log"
Jul 29 02:10:55.013601 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.013577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/thanos-query/0.log"
Jul 29 02:10:55.039297 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.039260    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/kube-rbac-proxy-web/0.log"
Jul 29 02:10:55.064949 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.064924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/kube-rbac-proxy/0.log"
Jul 29 02:10:55.094411 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.094386    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/prom-label-proxy/0.log"
Jul 29 02:10:55.123607 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.123578    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/kube-rbac-proxy-rules/0.log"
Jul 29 02:10:55.151490 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.151459    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b94868948-jk4kd_f4e6eff0-2779-4d31-8967-0139e359c218/kube-rbac-proxy-metrics/0.log"
Jul 29 02:10:55.514831 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.514792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"]
Jul 29 02:10:55.522353 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.522325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.535972 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.535938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"]
Jul 29 02:10:55.606742 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.606699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-proc\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.606933 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.606839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfzw\" (UniqueName: \"kubernetes.io/projected/e8bd75c4-5cca-40a8-a08d-ba60e3260211-kube-api-access-2rfzw\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.606933 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.606878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-lib-modules\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.607078 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.606956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-podres\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.607078 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.607004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-sys\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708539 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-proc\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfzw\" (UniqueName: \"kubernetes.io/projected/e8bd75c4-5cca-40a8-a08d-ba60e3260211-kube-api-access-2rfzw\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-proc\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-lib-modules\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-podres\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.708733 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-sys\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.709013 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-lib-modules\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.709013 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-sys\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.709013 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.708858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8bd75c4-5cca-40a8-a08d-ba60e3260211-podres\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.721460 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.721430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfzw\" (UniqueName: \"kubernetes.io/projected/e8bd75c4-5cca-40a8-a08d-ba60e3260211-kube-api-access-2rfzw\") pod \"perf-node-gather-daemonset-c6jpz\" (UID: \"e8bd75c4-5cca-40a8-a08d-ba60e3260211\") " pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:55.837303 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:55.837267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:56.018936 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:56.018895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"]
Jul 29 02:10:56.022538 ip-10-0-136-251 kubenswrapper[2577]: W0729 02:10:56.022504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8bd75c4_5cca_40a8_a08d_ba60e3260211.slice/crio-83b4fe9cbf0e83b941e11678f7adfba42f1ef6ddcac05efd01045c59048bfb49 WatchSource:0}: Error finding container 83b4fe9cbf0e83b941e11678f7adfba42f1ef6ddcac05efd01045c59048bfb49: Status 404 returned error can't find the container with id 83b4fe9cbf0e83b941e11678f7adfba42f1ef6ddcac05efd01045c59048bfb49
Jul 29 02:10:56.631322 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:56.631287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz" event={"ID":"e8bd75c4-5cca-40a8-a08d-ba60e3260211","Type":"ContainerStarted","Data":"7215f6d6620ff538a6aae471be18f35fa6560009c9c94c43099ace706417619a"}
Jul 29 02:10:56.631781 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:56.631341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz" event={"ID":"e8bd75c4-5cca-40a8-a08d-ba60e3260211","Type":"ContainerStarted","Data":"83b4fe9cbf0e83b941e11678f7adfba42f1ef6ddcac05efd01045c59048bfb49"}
Jul 29 02:10:56.631781 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:56.631433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:10:56.650835 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:56.650777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz" podStartSLOduration=1.650756363 podStartE2EDuration="1.650756363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:10:56.649490687 +0000 UTC m=+3643.802936064" watchObservedRunningTime="2026-07-29 02:10:56.650756363 +0000 UTC m=+3643.804201741"
Jul 29 02:10:57.531658 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:57.531628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-tpbnn_ff6acb4d-a8b5-48ce-88f3-f7b7a9a01a83/download-server/0.log"
Jul 29 02:10:59.279036 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:59.279004    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jq8v5_1e652714-955b-4d9d-a768-f781cf121cea/dns/0.log"
Jul 29 02:10:59.302872 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:59.302844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jq8v5_1e652714-955b-4d9d-a768-f781cf121cea/kube-rbac-proxy/0.log"
Jul 29 02:10:59.447494 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:10:59.447466    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsfjt_e84ac7e9-e8a9-4357-aefc-48f48a65d883/dns-node-resolver/0.log"
Jul 29 02:11:00.073025 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:00.072993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mfgmj_bae9cb32-dcf4-4464-b83f-77ad30c84c7a/node-ca/0.log"
Jul 29 02:11:01.384932 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:01.384905    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86657cdbcd-glh4b_7b46c7ae-7f5a-47ed-b546-84147b0f102d/kube-auth-proxy/0.log"
Jul 29 02:11:02.528091 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:02.528050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xnxd6_1fd6be0d-2df3-4faf-9974-ca1d237b2c33/serve-healthcheck-canary/0.log"
Jul 29 02:11:02.646135 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:02.646110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hd2h/perf-node-gather-daemonset-c6jpz"
Jul 29 02:11:03.070123 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:03.070099    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z2kk_c8fc14f1-4a24-4d26-94d4-4c814c404f22/kube-rbac-proxy/0.log"
Jul 29 02:11:03.098227 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:03.098201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z2kk_c8fc14f1-4a24-4d26-94d4-4c814c404f22/exporter/0.log"
Jul 29 02:11:03.126188 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:03.126152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z2kk_c8fc14f1-4a24-4d26-94d4-4c814c404f22/extractor/0.log"
Jul 29 02:11:06.617608 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:06.617576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tr5wm_d900390a-d129-4426-8525-5bbc998014da/server/0.log"
Jul 29 02:11:06.638633 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:06.638612    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86657cdbcd-glh4b_7b46c7ae-7f5a-47ed-b546-84147b0f102d/kube-auth-proxy/0.log"
Jul 29 02:11:07.036448 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:07.036422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d7cdd4c-l25bs_ff586751-2ef6-467b-9381-4b177ab766c3/manager/0.log"
Jul 29 02:11:07.065575 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:07.065544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d7cdd4c-l25bs_ff586751-2ef6-467b-9381-4b177ab766c3/copy-manifests/0.log"
Jul 29 02:11:08.307496 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:08.307468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c7f9c9bb4-ngq9z_58729e2a-366b-4866-a52f-82dfe3801548/manager/0.log"
Jul 29 02:11:08.450988 ip-10-0-136-251 kubenswrapper[2577]: I0729 02:11:08.450963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86657cdbcd-glh4b_7b46c7ae-7f5a-47ed-b546-84147b0f102d/kube-auth-proxy/0.log"