Sep 11 12:26:57.041594 ip-10-0-140-247 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 12:26:57.448273 ip-10-0-140-247 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:26:57.448273 ip-10-0-140-247 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 12:26:57.448273 ip-10-0-140-247 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:26:57.448273 ip-10-0-140-247 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 12:26:57.448273 ip-10-0-140-247 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:26:57.448969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.448411    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 12:26:57.452470 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452454    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452472    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452477    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452481    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452484    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452487    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452490    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452493    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452497    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452500    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452502    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452505    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452508    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452511    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.452507 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452514    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452517    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452519    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452522    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452525    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452527    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452530    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452532    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452535    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452543    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452546    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452548    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452550    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452553    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452556    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452558    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452561    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452563    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452566    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.452846 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452569    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452571    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452574    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452576    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452579    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452581    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452584    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452587    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452589    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452592    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452594    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452597    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452600    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452602    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452605    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452608    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452611    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452613    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452616    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452619    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.453315 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452621    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452624    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452627    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452629    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452632    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452635    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452638    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452640    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452643    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452646    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452648    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452651    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452654    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452656    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452659    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452661    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452664    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452666    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452669    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452672    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.453828 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452675    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452677    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452680    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452683    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452685    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452688    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452691    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452693    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452696    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452699    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452701    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452706    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452710    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452713    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452716    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452718    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452721    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452725    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452728    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.454333 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452731    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452734    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452737    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452739    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452742    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452745    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.452748    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453464    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453472    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453475    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453478    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453482    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453484    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453487    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453490    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453493    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453497    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453500    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453502    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.454807 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453506    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453509    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453512    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453515    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453518    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453520    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453523    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453526    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453529    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453531    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453534    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453536    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453539    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453542    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453545    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453548    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453550    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453553    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453556    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453558    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.455275 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453561    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453563    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453566    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453569    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453571    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453574    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453576    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453579    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453581    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453584    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453587    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453591    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453595    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453598    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453601    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453604    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453607    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453609    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453612    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.455784 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453615    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453617    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453620    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453623    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453625    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453628    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453630    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453633    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453636    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453638    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453641    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453643    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453646    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453648    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453651    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453654    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453658    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453661    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453664    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.456247 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453667    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453670    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453673    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453676    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453679    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453682    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453685    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453689    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453692    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453695    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453697    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453700    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453703    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453705    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453708    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453711    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453714    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453716    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453719    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453721    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.456758 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453724    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453726    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453730    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453734    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453737    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453740    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453742    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453745    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.453747    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454092    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454096    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454099    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454101    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454104    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454107    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454109    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454112    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454115    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454119    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454122    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.457260 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454125    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454128    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454132    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454136    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454138    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454141    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454144    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454147    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454149    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454152    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454155    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454157    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454159    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454162    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454165    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454167    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454171    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454173    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454176    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454178    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.457771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454181    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454184    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454187    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454189    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454192    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454194    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454197    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454199    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454201    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454204    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454208    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454210    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454213    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454215    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454218    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454221    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454223    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454226    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454229    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454232    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.458293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454234    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454236    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454239    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454242    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454244    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454247    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454249    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454252    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454254    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454258    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454260    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454263    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454265    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454268    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454270    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454273    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454275    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454278    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454280    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454283    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.458797 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454285    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454290    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454293    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454296    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454299    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454301    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454304    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454306    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454309    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454311    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454314    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454316    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454319    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454337    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454341    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454345    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454348    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454351    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454355    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.459274 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454358    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454361    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454364    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454367    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454370    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454373    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454375    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454378    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.454381    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454451    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454485    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454491    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454495    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454500    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454506    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454515    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454520    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454523    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454527    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454530    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454534    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454537    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 12:26:57.459780 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454540    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454543    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454546    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454548    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454551    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454556    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454559    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454562    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454566    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454569    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454572    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454575    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454578    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454582    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454585    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454588    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454591    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454594    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454597    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454601    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454604    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454607    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454610    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454613    2575 flags.go:64] FLAG: --enable-server="true"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454618    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 12:26:57.460322 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454622    2575 flags.go:64] FLAG: --event-burst="100"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454625    2575 flags.go:64] FLAG: --event-qps="50"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454628    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454632    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454635    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454639    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454641    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454644    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454647    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454650    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454653    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454657    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454659    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454662    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454665    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454668    2575 flags.go:64] FLAG: --feature-gates=""
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454672    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454674    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454677    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454680    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454683    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454687    2575 flags.go:64] FLAG: --help="false"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454690    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454693    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 12:26:57.460962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454695    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454698    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454702    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454705    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454708    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454711    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454713    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454718    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454721    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454724    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454727    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454729    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454733    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454736    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454739    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454742    2575 flags.go:64] FLAG: --lock-file=""
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454744    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454747    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454750    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454755    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454758    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454761    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454764    2575 flags.go:64] FLAG: --logging-format="text"
Sep 11 12:26:57.461584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454767    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454770    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454773    2575 flags.go:64] FLAG: --manifest-url=""
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454775    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454780    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454783    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454787    2575 flags.go:64] FLAG: --max-pods="110"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454790    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454793    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454796    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454799    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454802    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454805    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454808    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454815    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454818    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454823    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454826    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454829    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454835    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454838    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454841    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454844    2575 flags.go:64] FLAG: --port="10250"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454847    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 12:26:57.462169 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454850    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e55aef5256a3b6fb"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454853    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454856    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454859    2575 flags.go:64] FLAG: --register-node="true"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454862    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454866    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454869    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454871    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454874    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454878    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454881    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454884    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454887    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454889    2575 flags.go:64] FLAG: --runonce="false"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454892    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454896    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454899    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454902    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454905    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454908    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454911    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454915    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454918    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454921    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454925    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454929    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 12:26:57.462779 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454932    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454935    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454938    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454943    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454946    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454950    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454954    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454956    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454959    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454962    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454965    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454968    2575 flags.go:64] FLAG: --v="2"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454972    2575 flags.go:64] FLAG: --version="false"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454976    2575 flags.go:64] FLAG: --vmodule=""
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454980    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.454983    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455089    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455093    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455097    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455102    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455105    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455108    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455111    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.463467 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455114    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455117    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455120    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455124    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455126    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455129    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455132    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455136    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455140    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455143    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455146    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455149    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455151    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455154    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455157    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455160    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455163    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455165    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455168    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455171    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.464035 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455173    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455176    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455179    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455181    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455184    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455186    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455189    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455191    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455194    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455197    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455199    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455202    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455205    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455207    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455210    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455212    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455215    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455218    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455220    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455224    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.464546 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455228    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455231    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455233    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455236    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455239    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455241    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455244    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455247    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455250    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455253    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455255    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455258    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455260    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455263    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455266    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455268    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455271    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455273    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455276    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455279    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.465039 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455282    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455285    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455287    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455290    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455292    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455295    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455298    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455302    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455306    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455309    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455313    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455318    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455334    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455337    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455340    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455343    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455347    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455350    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455353    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.465530 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455355    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455358    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455361    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455364    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455367    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455369    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455373    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455377    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455380    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455383    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455386    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455388    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.455391    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.456096    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:26:57.466009 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.463095    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.463108    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.464593    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.464604    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464660    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464665    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464669    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464674    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464678    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464682    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464686    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464689    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464693    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464697    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464700    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464703    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464706    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464709    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464712    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.466426 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464715    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464718    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464721    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464725    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464728    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464732    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464735    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464738    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464741    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464743    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464746    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464750    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464754    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464758    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464763    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464766    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464769    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464772    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464774    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.466889 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464777    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464780    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464783    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464785    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464789    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464792    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464794    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464797    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464799    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464802    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464805    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464807    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464810    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464813    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464816    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464819    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464822    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464824    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464828    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464832    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.467377 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464836    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464840    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464844    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464846    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464849    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464852    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464854    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464857    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464860    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464863    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464865    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464868    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464871    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464873    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464876    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464879    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464881    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464884    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464886    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.467922 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464889    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464891    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464894    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464896    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464900    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464904    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464908    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464913    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464918    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464924    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464927    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464930    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464939    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464942    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464945    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464949    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464952    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464955    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464958    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464961    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.468511 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464964    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464967    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464970    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464972    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464975    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.464978    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.464983    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465104    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465109    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465112    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465115    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465118    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465121    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465123    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:26:57.469074 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465126    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465128    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465131    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465133    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465136    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465138    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465141    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465148    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465154    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465159    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465162    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465165    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465167    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465170    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465172    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465175    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465178    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465180    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465183    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:26:57.469432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465187    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465189    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465192    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465194    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465197    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465199    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465202    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465204    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465207    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465209    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465212    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465214    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465217    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465219    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465223    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465227    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465231    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465235    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465238    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465241    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465244    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:26:57.469962 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465246    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465249    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465252    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465254    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465257    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465259    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465262    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465265    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465267    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465270    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465273    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465276    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465279    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465282    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465285    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465287    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465290    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465293    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465295    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:26:57.470486 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465298    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465301    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465306    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465310    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465314    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465318    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465320    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465333    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465337    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465341    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465344    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465347    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465350    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465352    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465355    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465358    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465361    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465363    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465366    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:26:57.471041 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465368    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465371    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465373    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465376    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465378    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465381    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465384    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465386    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465389    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465391    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465394    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465397    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465401    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:57.465405    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.465413    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:26:57.471537 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.465506    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 12:26:57.472147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.467544    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 12:26:57.472147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.468590    2575 server.go:1019] "Starting client certificate rotation"
Sep 11 12:26:57.472147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.468708    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:26:57.472147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.469625    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:26:57.472147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.471022    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:26:57.472927 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.472908    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:26:57.486770 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.486752    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 11 12:26:57.489766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.489750    2575 log.go:25] "Validated CRI v1 image API"
Sep 11 12:26:57.490593 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.490577    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 12:26:57.494185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.494167    2575 fs.go:135] Filesystem UUIDs: map[7795c6ee-44b8-4211-ada9-ec20a3079926:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d17b8667-6db0-4d8b-8781-425b3bcb4831:/dev/nvme0n1p4]
Sep 11 12:26:57.494239 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.494184    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 12:26:57.498357 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.498342    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:26:57.499194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.499089    2575 manager.go:216] Machine: {Timestamp:2026-09-11 12:26:57.497707522 +0000 UTC m=+0.339458659 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106288 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216b6b3ff9c53ed1abca74b9d48b10 SystemUUID:ec216b6b-3ff9-c53e-d1ab-ca74b9d48b10 BootID:fd0fd1d1-504f-4396-b640-d34909eaf4c3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:f7:a6:14:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:f7:a6:14:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:3d:44:e7:3f:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 12:26:57.499194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.499194    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 12:26:57.499284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.499270    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 12:26:57.502746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.502726    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 12:26:57.502875 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.502749    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 12:26:57.503540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.503531    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 12:26:57.503575 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.503542    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 12:26:57.503575 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.503549    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:26:57.505027 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.505016    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:26:57.505809 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.505800    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 12:26:57.509139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509128    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:26:57.509236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509227    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 12:26:57.509270 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509255    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 12:26:57.509270 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509266    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 12:26:57.509338 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509279    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 12:26:57.509338 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509288    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 11 12:26:57.509338 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509297    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 12:26:57.509647 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509636    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:26:57.509689 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.509655    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:26:57.511698 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.511683    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 12:26:57.513635 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.513619    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 12:26:57.513701 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.513669    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 12:26:57.515215 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515203    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 12:26:57.515265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515220    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 12:26:57.515265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515226    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 12:26:57.515265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515231    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 12:26:57.515265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515239    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 12:26:57.515265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515256    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515270    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515275    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515281    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515293    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515304    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 12:26:57.515417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.515309    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 12:26:57.516040 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.516029    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 12:26:57.516040 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.516038    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 12:26:57.519339 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.519303    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 12:26:57.519339 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.519306    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 12:26:57.519339 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.519339    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-247.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 12:26:57.519726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.519714    2575 server.go:1268] "Started kubelet"
Sep 11 12:26:57.520063 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.519996    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 12:26:57.520223 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.520192    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 12:26:57.520407 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.520348    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 12:26:57.520790 ip-10-0-140-247 systemd[1]: Started Kubernetes Kubelet.
Sep 11 12:26:57.525343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.525302    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 12:26:57.526916 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.526893    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 12:26:57.527064 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.527038    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 12:26:57.530007 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.529987    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hrrps"
Sep 11 12:26:57.532012 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.530990    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-247.ec2.internal.18d443e86d643a6d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-247.ec2.internal,UID:ip-10-0-140-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-247.ec2.internal,},FirstTimestamp:2026-09-11 12:26:57.519688301 +0000 UTC m=+0.361439419,LastTimestamp:2026-09-11 12:26:57.519688301 +0000 UTC m=+0.361439419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-247.ec2.internal,}"
Sep 11 12:26:57.532012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.531995    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 12:26:57.532181 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532001    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 12:26:57.532181 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.532102    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:57.532600 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532582    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 12:26:57.532600 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532590    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 12:26:57.532738 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532611    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 12:26:57.532738 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532731    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 12:26:57.532803 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532770    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 12:26:57.532966 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532946    2575 factory.go:55] Registering systemd factory
Sep 11 12:26:57.532966 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.532966    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 11 12:26:57.533207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533193    2575 factory.go:153] Registering CRI-O factory
Sep 11 12:26:57.533290 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533210    2575 factory.go:223] Registration of the crio container factory successfully
Sep 11 12:26:57.533290 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533274    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 12:26:57.533416 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533293    2575 factory.go:103] Registering Raw factory
Sep 11 12:26:57.533416 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533308    2575 manager.go:1196] Started watching for new ooms in manager
Sep 11 12:26:57.533650 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.533637    2575 manager.go:349] Starting recovery of all containers
Sep 11 12:26:57.536979 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.536959    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hrrps"
Sep 11 12:26:57.543955 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.543847    2575 manager.go:354] Recovery completed
Sep 11 12:26:57.545285 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.543840    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 12:26:57.545285 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.543929    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 12:26:57.550461 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.550448    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.552921 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.552906    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.553020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.552939    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.553020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.552954    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.553583 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.553567    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 12:26:57.553583 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.553583    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 12:26:57.553713 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.553600    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:26:57.553713 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.553584    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.554377 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554360    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.554446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554390    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.554446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554399    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.554807 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554795    2575 policy_none.go:49] "None policy: Start"
Sep 11 12:26:57.554837 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554813    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 12:26:57.554837 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.554823    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 12:26:57.555927 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.555916    2575 policy_none.go:47] "Start"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.591880    2575 manager.go:337] "Starting Device Plugin manager"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.591917    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.591930    2575 server.go:87] "Starting device plugin registration server"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.592216    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.592227    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.592357    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.592446    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.592456    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.592926    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 12:26:57.603729 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.592974    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:57.622130 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.622103    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 12:26:57.623337 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.623304    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 12:26:57.623427 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.623373    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 12:26:57.623427 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.623400    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 12:26:57.623530 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.623480    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 12:26:57.625339 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.625307    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:26:57.693212 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.693188    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.694176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.694160    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.694263 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.694189    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.694263 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.694199    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.694263 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.694222    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.701914 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.701857    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.701914 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.701883    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-247.ec2.internal\": node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:57.719738 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.719717    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:57.723776 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.723759    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"]
Sep 11 12:26:57.723827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.723819    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.725181 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.725167    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.725252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.725198    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.725252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.725209    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.726573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.726562    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.726688 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.726674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.726746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.726702    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.727258 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727244    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.727343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727263    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.727343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727288    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.727343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727302    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.727343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727270    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.727528 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.727368    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.728879 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.728863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.728945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.728893    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:26:57.729549 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.729536    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:26:57.729602 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.729563    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:26:57.729602 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.729572    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:26:57.733184 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.733162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.733283 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.733192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.733283 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.733207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6f1705d022cd6d1f69ce3536d23a7e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-247.ec2.internal\" (UID: \"8e6f1705d022cd6d1f69ce3536d23a7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.755034 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.755016    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-247.ec2.internal\" not found" node="ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.759613 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.759593    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-247.ec2.internal\" not found" node="ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.820784 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.820768    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:57.834035 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.834120 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.834120 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6f1705d022cd6d1f69ce3536d23a7e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-247.ec2.internal\" (UID: \"8e6f1705d022cd6d1f69ce3536d23a7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.834193 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e6f1705d022cd6d1f69ce3536d23a7e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-247.ec2.internal\" (UID: \"8e6f1705d022cd6d1f69ce3536d23a7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.834193 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.834193 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:57.834149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2b052881250fe4c901d0fc8ea0f362e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal\" (UID: \"e2b052881250fe4c901d0fc8ea0f362e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:57.921202 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:57.921181    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.022302 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.022233    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.057739 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.057721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:58.062745 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.062730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:58.123301 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.123276    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.223665 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.223635    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.330232 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.330169    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.432380 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.432323    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.468373 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.468342    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 12:26:58.468807 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.468510    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 12:26:58.532873 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.532825    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.545553 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.545523    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 12:21:57 +0000 UTC" deadline="2028-02-21 09:16:15.915801779 +0000 UTC"
Sep 11 12:26:58.545553 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.545550    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12668h49m17.370254598s"
Sep 11 12:26:58.633718 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.633693    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.733858 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.733832    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.733858 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:58.733831    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b052881250fe4c901d0fc8ea0f362e.slice/crio-88f8e8fdfa23c80fa714964ae6ec70025b97a116e60f117384cf5f0ddf711447 WatchSource:0}: Error finding container 88f8e8fdfa23c80fa714964ae6ec70025b97a116e60f117384cf5f0ddf711447: Status 404 returned error can't find the container with id 88f8e8fdfa23c80fa714964ae6ec70025b97a116e60f117384cf5f0ddf711447
Sep 11 12:26:58.734049 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:26:58.734031    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6f1705d022cd6d1f69ce3536d23a7e.slice/crio-43a2fb8751aaed3b19297bffe561cfd345548a0698fc84a4236fe0d6a7f0c241 WatchSource:0}: Error finding container 43a2fb8751aaed3b19297bffe561cfd345548a0698fc84a4236fe0d6a7f0c241: Status 404 returned error can't find the container with id 43a2fb8751aaed3b19297bffe561cfd345548a0698fc84a4236fe0d6a7f0c241
Sep 11 12:26:58.741270 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.741252    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:26:58.834031 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.833989    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:58.911513 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:58.911422    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:26:58.934624 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:58.934589    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:59.034810 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.034788    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-247.ec2.internal\" not found"
Sep 11 12:26:59.037049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.037031    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:26:59.056128 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.056105    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:26:59.133584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.133544    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:59.146271 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.146248    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:26:59.147293 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.147279    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal"
Sep 11 12:26:59.156937 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.156909    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:26:59.510392 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.510355    2575 apiserver.go:52] "Watching apiserver"
Sep 11 12:26:59.520375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.520342    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 12:26:59.521588 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.521563    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn","openshift-cluster-node-tuning-operator/tuned-gt7b4","openshift-dns/node-resolver-5sw85","openshift-image-registry/node-ca-t48d7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal","openshift-multus/multus-9hjxt","openshift-multus/multus-additional-cni-plugins-9b6k9","openshift-multus/network-metrics-daemon-mfbtn","kube-system/global-pull-secret-syncer-mzvww","kube-system/konnectivity-agent-tx7bj","kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal","openshift-network-diagnostics/network-check-target-n94wh","openshift-network-operator/iptables-alerter-45btd","openshift-ovn-kubernetes/ovnkube-node-cqn98"]
Sep 11 12:26:59.523212 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.523189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.524387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.524348    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.525798 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.525768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.526456 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.526434    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-27wvb\""
Sep 11 12:26:59.526566 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.526465    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 12:26:59.526666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.526652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 12:26:59.527093 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.527067    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.527203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.527181    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 12:26:59.527269 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.527187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:26:59.527325 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.527267    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.527325 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.527273    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:26:59.527471 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.527451    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nj78d\""
Sep 11 12:26:59.528833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.528515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f9dfj\""
Sep 11 12:26:59.528833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.528582    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.528833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.528612    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 12:26:59.528833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.528656    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.529121 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.528965    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.529121 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.529026    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:26:59.530421 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.530386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.532087 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.532014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.532946 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.532926    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 12:26:59.533562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.533188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.533562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.533264    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.533562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.533404    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jgwhk\""
Sep 11 12:26:59.533562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.533556    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.534686 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.534665    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.534788 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.534732    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.534788 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.534743    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:26:59.534963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.534944    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 12:26:59.535036 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.534967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.535092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.535046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 12:26:59.535092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.535063    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r8mmf\""
Sep 11 12:26:59.535582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.535522    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 12:26:59.535833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.535796    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 12:26:59.536191 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.536170    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7zj2q\""
Sep 11 12:26:59.536427 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.536409    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.537908 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.537888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.539442 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.539419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.539554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.539531    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mdl68\""
Sep 11 12:26:59.539630 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.539552    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.540042 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.540020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.540565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.540547    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.541374 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.541175    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 12:26:59.541374 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.541206    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.541648 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.541628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7ftm\""
Sep 11 12:26:59.542566 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.542540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cl7z4\""
Sep 11 12:26:59.542668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.542635    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 12:26:59.542749 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.542683    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 12:26:59.542894 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.542878    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 12:26:59.542987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.542936    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 12:26:59.543139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.543116    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 12:26:59.543419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.543398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-bin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-multus-certs\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-serviceca\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545185    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-cni-binary-copy\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-multus\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-daemon-config\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-hostroot\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clj9h\" (UniqueName: \"kubernetes.io/projected/61a373b3-a7f6-4108-bd1d-604a51547fb3-kube-api-access-clj9h\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-host\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jgs\" (UniqueName: \"kubernetes.io/projected/79a19085-260b-448b-b9b6-abb6affcda26-kube-api-access-66jgs\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-conf-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.547540 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebfa985d-6c0e-4bea-be94-078f3bef8d04-konnectivity-ca\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9hn\" (UniqueName: \"kubernetes.io/projected/50f30488-bba7-4cb1-83ec-e9e58eb6000e-kube-api-access-zp9hn\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a19085-260b-448b-b9b6-abb6affcda26-iptables-alerter-script\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545683    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-socket-dir-parent\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfbj\" (UniqueName: \"kubernetes.io/projected/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-kube-api-access-6pfbj\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcpp\" (UniqueName: \"kubernetes.io/projected/08825d8c-0fb4-41a5-9624-0d87e65963a4-kube-api-access-mzcpp\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-system-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-cnibin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-kubelet\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-etc-kubernetes\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de85618e-7d30-44b9-87f0-dda082e4718e-hosts-file\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.545992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de85618e-7d30-44b9-87f0-dda082e4718e-tmp-dir\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-k8s-cni-cncf-io\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-netns\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebfa985d-6c0e-4bea-be94-078f3bef8d04-agent-certs\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.548305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.549204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a19085-260b-448b-b9b6-abb6affcda26-host-slash\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.549204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:26:59.549204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzm2t\" (UniqueName: \"kubernetes.io/projected/de85618e-7d30-44b9-87f0-dda082e4718e-kube-api-access-hzm2t\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.549204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-system-cni-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.549204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.546787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cnibin\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.549636 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.549616    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:26:59.575207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.575183    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxm2p"
Sep 11 12:26:59.584680 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.584663    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxm2p"
Sep 11 12:26:59.628744 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.628696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal" event={"ID":"8e6f1705d022cd6d1f69ce3536d23a7e","Type":"ContainerStarted","Data":"43a2fb8751aaed3b19297bffe561cfd345548a0698fc84a4236fe0d6a7f0c241"}
Sep 11 12:26:59.629981 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.629957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal" event={"ID":"e2b052881250fe4c901d0fc8ea0f362e","Type":"ContainerStarted","Data":"88f8e8fdfa23c80fa714964ae6ec70025b97a116e60f117384cf5f0ddf711447"}
Sep 11 12:26:59.633916 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.633896    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 12:26:59.647050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rsl\" (UniqueName: \"kubernetes.io/projected/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-kube-api-access-t6rsl\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647156 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-socket-dir-parent\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647156 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfbj\" (UniqueName: \"kubernetes.io/projected/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-kube-api-access-6pfbj\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.647275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcpp\" (UniqueName: \"kubernetes.io/projected/08825d8c-0fb4-41a5-9624-0d87e65963a4-kube-api-access-mzcpp\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.647275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-conf\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-socket-dir-parent\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-systemd\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-script-lib\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-system-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-cnibin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de85618e-7d30-44b9-87f0-dda082e4718e-hosts-file\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysconfig\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-kubernetes\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-cnibin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-lib-modules\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-system-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de85618e-7d30-44b9-87f0-dda082e4718e-hosts-file\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-socket-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-k8s-cni-cncf-io\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebfa985d-6c0e-4bea-be94-078f3bef8d04-agent-certs\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647731    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-k8s-cni-cncf-io\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-registration-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-ovn\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovn-node-metrics-cert\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cnibin\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-serviceca\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.647943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-var-lib-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647969    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.647997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-cni-binary-copy\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-multus\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-daemon-config\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-kubelet-config\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-hostroot\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-sys-fs\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-systemd-units\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648183    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-systemd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-netd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-conf-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebfa985d-6c0e-4bea-be94-078f3bef8d04-konnectivity-ca\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648345    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648366    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cnibin\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.648781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9hn\" (UniqueName: \"kubernetes.io/projected/50f30488-bba7-4cb1-83ec-e9e58eb6000e-kube-api-access-zp9hn\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a19085-260b-448b-b9b6-abb6affcda26-iptables-alerter-script\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-tuned\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-node-log\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6qp\" (UniqueName: \"kubernetes.io/projected/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kube-api-access-xn6qp\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-kubelet\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-hostroot\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-serviceca\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-multus\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-config\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-conf-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-cni-dir\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-kubelet\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-etc-kubernetes\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-kubelet\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.649624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-etc-kubernetes\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de85618e-7d30-44b9-87f0-dda082e4718e-tmp-dir\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-device-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.648980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-multus-daemon-config\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-etc-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-netns\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a19085-260b-448b-b9b6-abb6affcda26-host-slash\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzm2t\" (UniqueName: \"kubernetes.io/projected/de85618e-7d30-44b9-87f0-dda082e4718e-kube-api-access-hzm2t\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fztv\" (UniqueName: \"kubernetes.io/projected/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-kube-api-access-4fztv\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-netns\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-log-socket\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649189    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-bin\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-system-cni-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a19085-260b-448b-b9b6-abb6affcda26-iptables-alerter-script\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-modprobe-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61a373b3-a7f6-4108-bd1d-604a51547fb3-cni-binary-copy\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-bin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.650415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebfa985d-6c0e-4bea-be94-078f3bef8d04-konnectivity-ca\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649283    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a19085-260b-448b-b9b6-abb6affcda26-host-slash\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de85618e-7d30-44b9-87f0-dda082e4718e-tmp-dir\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-multus-certs\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-system-cni-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-var-lib-cni-bin\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-host\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-tmp\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61a373b3-a7f6-4108-bd1d-604a51547fb3-host-run-multus-certs\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-slash\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-dbus\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-var-lib-kubelet\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clj9h\" (UniqueName: \"kubernetes.io/projected/61a373b3-a7f6-4108-bd1d-604a51547fb3-kube-api-access-clj9h\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-host\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jgs\" (UniqueName: \"kubernetes.io/projected/79a19085-260b-448b-b9b6-abb6affcda26-kube-api-access-66jgs\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.651096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-host\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-netns\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649849    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50f30488-bba7-4cb1-83ec-e9e58eb6000e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-run\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.649900    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-sys\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-env-overrides\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.649974    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:00.14995231 +0000 UTC m=+2.991703425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:26:59.651965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.649806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f30488-bba7-4cb1-83ec-e9e58eb6000e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.653369 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.653337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebfa985d-6c0e-4bea-be94-078f3bef8d04-agent-certs\") pod \"konnectivity-agent-tx7bj\" (UID: \"ebfa985d-6c0e-4bea-be94-078f3bef8d04\") " pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.662307 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.662286    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:26:59.662418 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.662313    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:26:59.662418 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.662327    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:26:59.662418 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.662391    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:00.162371021 +0000 UTC m=+3.004122149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:26:59.663054 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.663022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcpp\" (UniqueName: \"kubernetes.io/projected/08825d8c-0fb4-41a5-9624-0d87e65963a4-kube-api-access-mzcpp\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:26:59.663500 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.663475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9hn\" (UniqueName: \"kubernetes.io/projected/50f30488-bba7-4cb1-83ec-e9e58eb6000e-kube-api-access-zp9hn\") pod \"multus-additional-cni-plugins-9b6k9\" (UID: \"50f30488-bba7-4cb1-83ec-e9e58eb6000e\") " pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.663588 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.663511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzm2t\" (UniqueName: \"kubernetes.io/projected/de85618e-7d30-44b9-87f0-dda082e4718e-kube-api-access-hzm2t\") pod \"node-resolver-5sw85\" (UID: \"de85618e-7d30-44b9-87f0-dda082e4718e\") " pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.664162 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.664133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfbj\" (UniqueName: \"kubernetes.io/projected/8dd90914-8e03-4ea0-bfe8-bd7a31341a37-kube-api-access-6pfbj\") pod \"node-ca-t48d7\" (UID: \"8dd90914-8e03-4ea0-bfe8-bd7a31341a37\") " pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.664329 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.663862    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clj9h\" (UniqueName: \"kubernetes.io/projected/61a373b3-a7f6-4108-bd1d-604a51547fb3-kube-api-access-clj9h\") pod \"multus-9hjxt\" (UID: \"61a373b3-a7f6-4108-bd1d-604a51547fb3\") " pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.665756 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.665731    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jgs\" (UniqueName: \"kubernetes.io/projected/79a19085-260b-448b-b9b6-abb6affcda26-kube-api-access-66jgs\") pod \"iptables-alerter-45btd\" (UID: \"79a19085-260b-448b-b9b6-abb6affcda26\") " pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.750363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-host\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-tmp\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-slash\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-dbus\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-var-lib-kubelet\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-host\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-var-lib-kubelet\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-netns\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-dbus\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-slash\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-run\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-sys\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-netns\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-env-overrides\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rsl\" (UniqueName: \"kubernetes.io/projected/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-kube-api-access-t6rsl\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750609    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-run\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.750618    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-conf\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-systemd\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:26:59.750687    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:00.250667121 +0000 UTC m=+3.092418242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-sys\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-systemd\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-script-lib\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysconfig\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.750954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-conf\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-kubernetes\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-lib-modules\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysconfig\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-socket-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-kubernetes\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.750970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-registration-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-ovn\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751049    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-lib-modules\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovn-node-metrics-cert\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-registration-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-ovn\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-socket-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-env-overrides\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.751823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-var-lib-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751235    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-sysctl-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-script-lib\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751319    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751384    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-var-lib-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-kubelet-config\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751426    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-sys-fs\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-systemd-units\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-systemd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-netd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-systemd-units\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-sys-fs\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-run-systemd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-tuned\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751552    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74dbc65a-330f-406e-9f26-34cd5e77565d-kubelet-config\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:26:59.752476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751556    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-netd\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-node-log\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6qp\" (UniqueName: \"kubernetes.io/projected/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kube-api-access-xn6qp\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-kubelet\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751645    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-config\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-node-log\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-kubelet\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-device-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-etc-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fztv\" (UniqueName: \"kubernetes.io/projected/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-kube-api-access-4fztv\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-log-socket\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-bin\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49f375d-02d0-479f-8a69-03c9e7ebc8df-device-dir\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-modprobe-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-etc-openvswitch\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.752998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-modprobe-d\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.753573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751949    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-host-cni-bin\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.753573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.751965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-log-socket\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.753573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.752171    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovnkube-config\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.753573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.752816    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-tmp\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.753573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.753399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-ovn-node-metrics-cert\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.753769 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.753752    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-etc-tuned\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.760318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.760300    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rsl\" (UniqueName: \"kubernetes.io/projected/c8b267d8-27ee-4dca-b5bb-7149c3aeaf11-kube-api-access-t6rsl\") pod \"ovnkube-node-cqn98\" (UID: \"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:26:59.761636 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.761589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6qp\" (UniqueName: \"kubernetes.io/projected/e49f375d-02d0-479f-8a69-03c9e7ebc8df-kube-api-access-xn6qp\") pod \"aws-ebs-csi-driver-node-pnztn\" (UID: \"e49f375d-02d0-479f-8a69-03c9e7ebc8df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.761738 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.761722    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fztv\" (UniqueName: \"kubernetes.io/projected/d88cdc2f-2301-4f33-82cf-c3a8459f29ff-kube-api-access-4fztv\") pod \"tuned-gt7b4\" (UID: \"d88cdc2f-2301-4f33-82cf-c3a8459f29ff\") " pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.836447 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.836415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:26:59.845422 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.845399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t48d7"
Sep 11 12:26:59.854060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.854032    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45btd"
Sep 11 12:26:59.860722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.860697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5sw85"
Sep 11 12:26:59.870363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.870343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9b6k9"
Sep 11 12:26:59.877272 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.877251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9hjxt"
Sep 11 12:26:59.883935 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.883916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gt7b4"
Sep 11 12:26:59.896358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.896313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn"
Sep 11 12:26:59.896358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.896342    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:26:59.902987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:26:59.902970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:00.155671 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.155603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:00.155812 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.155743    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:00.155812 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.155802    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:01.155785632 +0000 UTC m=+3.997536736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:00.169552 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.169521    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfa985d_6c0e_4bea_be94_078f3bef8d04.slice/crio-11c1f97423adc738f30692a326015c8fab388b3a3c48e0f5cd7cc541965db4d8 WatchSource:0}: Error finding container 11c1f97423adc738f30692a326015c8fab388b3a3c48e0f5cd7cc541965db4d8: Status 404 returned error can't find the container with id 11c1f97423adc738f30692a326015c8fab388b3a3c48e0f5cd7cc541965db4d8
Sep 11 12:27:00.170693 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.170604    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f375d_02d0_479f_8a69_03c9e7ebc8df.slice/crio-50e2c97ee967b960162088ae86d62c80402c0a5690c8ff35982d7fc2fd0ac338 WatchSource:0}: Error finding container 50e2c97ee967b960162088ae86d62c80402c0a5690c8ff35982d7fc2fd0ac338: Status 404 returned error can't find the container with id 50e2c97ee967b960162088ae86d62c80402c0a5690c8ff35982d7fc2fd0ac338
Sep 11 12:27:00.171681 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.171615    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a373b3_a7f6_4108_bd1d_604a51547fb3.slice/crio-af856f60d724ef500f9a42ac5845720aeb24e1cce4bdab126f6b9f4c18d19ac4 WatchSource:0}: Error finding container af856f60d724ef500f9a42ac5845720aeb24e1cce4bdab126f6b9f4c18d19ac4: Status 404 returned error can't find the container with id af856f60d724ef500f9a42ac5845720aeb24e1cce4bdab126f6b9f4c18d19ac4
Sep 11 12:27:00.172308 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.172187    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde85618e_7d30_44b9_87f0_dda082e4718e.slice/crio-399879889c83c76fd2a5092ab5cc825bdeaf594c58315b452a8cba903fcfab2b WatchSource:0}: Error finding container 399879889c83c76fd2a5092ab5cc825bdeaf594c58315b452a8cba903fcfab2b: Status 404 returned error can't find the container with id 399879889c83c76fd2a5092ab5cc825bdeaf594c58315b452a8cba903fcfab2b
Sep 11 12:27:00.175909 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.175887    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a19085_260b_448b_b9b6_abb6affcda26.slice/crio-82dc93b73e1ccebdd98b6e8a2dffd9eb85e06437d650d14e557a8bbab2565e91 WatchSource:0}: Error finding container 82dc93b73e1ccebdd98b6e8a2dffd9eb85e06437d650d14e557a8bbab2565e91: Status 404 returned error can't find the container with id 82dc93b73e1ccebdd98b6e8a2dffd9eb85e06437d650d14e557a8bbab2565e91
Sep 11 12:27:00.176871 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.176852    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88cdc2f_2301_4f33_82cf_c3a8459f29ff.slice/crio-f8e55c01c474b29c3274c356275c115d20c10ba3dd8e4da726579163761f2989 WatchSource:0}: Error finding container f8e55c01c474b29c3274c356275c115d20c10ba3dd8e4da726579163761f2989: Status 404 returned error can't find the container with id f8e55c01c474b29c3274c356275c115d20c10ba3dd8e4da726579163761f2989
Sep 11 12:27:00.178652 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.178625    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f30488_bba7_4cb1_83ec_e9e58eb6000e.slice/crio-a003de753b0aad54e44472b9dab1006c13cb43c6daeb1b6eb0644116bdbb70cc WatchSource:0}: Error finding container a003de753b0aad54e44472b9dab1006c13cb43c6daeb1b6eb0644116bdbb70cc: Status 404 returned error can't find the container with id a003de753b0aad54e44472b9dab1006c13cb43c6daeb1b6eb0644116bdbb70cc
Sep 11 12:27:00.179953 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.179860    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b267d8_27ee_4dca_b5bb_7149c3aeaf11.slice/crio-dfad46245e6766093954de5fd8e534ba1a2788575835e81326782fe1c27ad9ca WatchSource:0}: Error finding container dfad46245e6766093954de5fd8e534ba1a2788575835e81326782fe1c27ad9ca: Status 404 returned error can't find the container with id dfad46245e6766093954de5fd8e534ba1a2788575835e81326782fe1c27ad9ca
Sep 11 12:27:00.180529 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:00.180516    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd90914_8e03_4ea0_bfe8_bd7a31341a37.slice/crio-3c8b6db531fb52ef242d8e11a26ce3b6e74a92bb78a0475181efe495ee273560 WatchSource:0}: Error finding container 3c8b6db531fb52ef242d8e11a26ce3b6e74a92bb78a0475181efe495ee273560: Status 404 returned error can't find the container with id 3c8b6db531fb52ef242d8e11a26ce3b6e74a92bb78a0475181efe495ee273560
Sep 11 12:27:00.257077 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.256899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:00.257172 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.257106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:00.257172 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257050    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:00.257259 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:01.257186374 +0000 UTC m=+4.098937480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:00.257259 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257227    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:00.257259 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257243    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:00.257259 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257256    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:00.257418 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.257319    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:01.257304545 +0000 UTC m=+4.099055651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:00.587480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.587409    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:21:59 +0000 UTC" deadline="2028-06-06 15:38:33.17866475 +0000 UTC"
Sep 11 12:27:00.587480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.587477    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15219h11m32.591192012s"
Sep 11 12:27:00.624055 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.624024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:00.624290 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.624242    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:00.624603 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.624439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:00.624603 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:00.624529    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:00.635310 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.635274    2575 generic.go:356] "Generic (PLEG): container finished" podID="e2b052881250fe4c901d0fc8ea0f362e" containerID="411177e7f7a2ac1f0532ee61ba5a6a4e007d2f12c7519b93421fb539944e788a" exitCode=0
Sep 11 12:27:00.635592 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.635565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal" event={"ID":"e2b052881250fe4c901d0fc8ea0f362e","Type":"ContainerDied","Data":"411177e7f7a2ac1f0532ee61ba5a6a4e007d2f12c7519b93421fb539944e788a"}
Sep 11 12:27:00.637379 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.637354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45btd" event={"ID":"79a19085-260b-448b-b9b6-abb6affcda26","Type":"ContainerStarted","Data":"82dc93b73e1ccebdd98b6e8a2dffd9eb85e06437d650d14e557a8bbab2565e91"}
Sep 11 12:27:00.640413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.640384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5sw85" event={"ID":"de85618e-7d30-44b9-87f0-dda082e4718e","Type":"ContainerStarted","Data":"399879889c83c76fd2a5092ab5cc825bdeaf594c58315b452a8cba903fcfab2b"}
Sep 11 12:27:00.642188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.642148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9hjxt" event={"ID":"61a373b3-a7f6-4108-bd1d-604a51547fb3","Type":"ContainerStarted","Data":"af856f60d724ef500f9a42ac5845720aeb24e1cce4bdab126f6b9f4c18d19ac4"}
Sep 11 12:27:00.644802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.644728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gt7b4" event={"ID":"d88cdc2f-2301-4f33-82cf-c3a8459f29ff","Type":"ContainerStarted","Data":"f8e55c01c474b29c3274c356275c115d20c10ba3dd8e4da726579163761f2989"}
Sep 11 12:27:00.646957 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.646928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"dfad46245e6766093954de5fd8e534ba1a2788575835e81326782fe1c27ad9ca"}
Sep 11 12:27:00.649506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.649456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerStarted","Data":"a003de753b0aad54e44472b9dab1006c13cb43c6daeb1b6eb0644116bdbb70cc"}
Sep 11 12:27:00.651726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.651698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t48d7" event={"ID":"8dd90914-8e03-4ea0-bfe8-bd7a31341a37","Type":"ContainerStarted","Data":"3c8b6db531fb52ef242d8e11a26ce3b6e74a92bb78a0475181efe495ee273560"}
Sep 11 12:27:00.653279 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.653252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn" event={"ID":"e49f375d-02d0-479f-8a69-03c9e7ebc8df","Type":"ContainerStarted","Data":"50e2c97ee967b960162088ae86d62c80402c0a5690c8ff35982d7fc2fd0ac338"}
Sep 11 12:27:00.655471 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:00.655415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tx7bj" event={"ID":"ebfa985d-6c0e-4bea-be94-078f3bef8d04","Type":"ContainerStarted","Data":"11c1f97423adc738f30692a326015c8fab388b3a3c48e0f5cd7cc541965db4d8"}
Sep 11 12:27:01.165806 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.165708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:01.165978 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.165912    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:01.165978 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.165978    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:03.165959096 +0000 UTC m=+6.007710205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:01.266895 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.266849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:01.267091 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.266924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:01.267153 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267088    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:01.267153 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267108    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:01.267153 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267122    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:01.267325 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267182    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:03.267162268 +0000 UTC m=+6.108913390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:01.267726 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267616    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:01.267726 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.267670    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:03.267654441 +0000 UTC m=+6.109405548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:01.588195 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.588151    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:21:59 +0000 UTC" deadline="2028-04-07 16:26:02.098876277 +0000 UTC"
Sep 11 12:27:01.588195 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.588193    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13779h59m0.510688423s"
Sep 11 12:27:01.626529 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.626501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:01.626690 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:01.626629    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:01.680425 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:01.679697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal" event={"ID":"e2b052881250fe4c901d0fc8ea0f362e","Type":"ContainerStarted","Data":"cd46d539db9f2e72336de02a5e5cabdac0f8c4ff5cd4eb3b7b3a38e32e5023f3"}
Sep 11 12:27:02.624860 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:02.623917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:02.624860 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:02.624055    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:02.624860 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:02.624721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:02.624860 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:02.624814    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:03.188989 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:03.188945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:03.189181 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.189162    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:03.189285 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.189233    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:07.189214601 +0000 UTC m=+10.030965711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:03.289739 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:03.289705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:03.289899 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:03.289766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:03.289899 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.289872    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:03.290024 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.289938    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:03.290024 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.289954    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:07.28993493 +0000 UTC m=+10.131686039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:03.290024 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.289960    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:03.290024 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.289974    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:03.290283 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.290020    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:07.290005819 +0000 UTC m=+10.131756929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:03.629416 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:03.626565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:03.629416 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:03.626686    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:04.624394 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:04.624341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:04.624566 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:04.624484    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:04.624917 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:04.624883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:04.624999 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:04.624986    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:05.624478 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:05.624392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:05.624965 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:05.624512    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:06.623778 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:06.623744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:06.623955 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:06.623788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:06.623955 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:06.623884    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:06.624193 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:06.624157    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:07.220766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:07.220726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:07.221372 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.220933    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:07.221372 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.221002    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:15.220980706 +0000 UTC m=+18.062731829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:07.322032 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:07.321967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:07.322032 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:07.322030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:07.322275 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322175    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:07.322275 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322193    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:07.322275 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322205    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:07.322275 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322207    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:07.322275 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322264    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:15.322246174 +0000 UTC m=+18.163997298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:07.322535 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.322294    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:15.3222738 +0000 UTC m=+18.164024909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:07.627257 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:07.627014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:07.627257 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:07.627152    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:08.624166 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:08.624119    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:08.624166 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:08.624157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:08.624669 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:08.624254    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:08.624669 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:08.624383    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:09.623702 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:09.623627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:09.623997 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:09.623758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:10.623790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:10.623756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:10.623790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:10.623777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:10.624222 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:10.623861    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:10.624222 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:10.624197    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:11.623816 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:11.623783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:11.624241 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:11.623904    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:12.624308 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:12.624274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:12.624308 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:12.624297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:12.624829 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:12.624410    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:12.624829 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:12.624490    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:13.623914 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:13.623878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:13.624077 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:13.623978    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:14.624414 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:14.624386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:14.624822 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:14.624491    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:14.624822 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:14.624546    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:14.624822 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:14.624625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:15.290032 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:15.289984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:15.290213 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.290122    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:15.290213 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.290198    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:27:31.290177589 +0000 UTC m=+34.131928707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:15.390586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:15.390549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:15.390749 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:15.390599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:15.390749 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390710    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:15.390749 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390727    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:15.390901 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390752    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:15.390901 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390768    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:15.390901 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390788    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:31.390770679 +0000 UTC m=+34.232521791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:15.390901 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.390819    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:27:31.390801455 +0000 UTC m=+34.232552590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:15.623671 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:15.623589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:15.623817 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:15.623717    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:16.624390 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:16.624352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:16.624790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:16.624369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:16.624790 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:16.624487    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:16.624790 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:16.624524    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:17.624932 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:17.624904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:17.625382 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:17.625003    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:18.624010 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.623826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:18.624010 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:18.623943    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:18.624143 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.624010    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:18.624143 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:18.624115    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:18.716515 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.716285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal" event={"ID":"8e6f1705d022cd6d1f69ce3536d23a7e","Type":"ContainerStarted","Data":"1484aef2e486ca025c34e4f418c6cc0a741966fc795f62fa1819a045d710c1cd"}
Sep 11 12:27:18.717986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.717964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5sw85" event={"ID":"de85618e-7d30-44b9-87f0-dda082e4718e","Type":"ContainerStarted","Data":"f9ffd60fa5d5c4e8a18436c662137c903a76b0bf8fe4e65af1e97d2f8ff52dd5"}
Sep 11 12:27:18.719926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.719899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9hjxt" event={"ID":"61a373b3-a7f6-4108-bd1d-604a51547fb3","Type":"ContainerStarted","Data":"a8daea42f8d2783f733f4230e3df0d0c16bb9d83e16a0ca4293b6afa7ef3cee2"}
Sep 11 12:27:18.721499 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.721472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gt7b4" event={"ID":"d88cdc2f-2301-4f33-82cf-c3a8459f29ff","Type":"ContainerStarted","Data":"e0cdfd64f4b11fee3fd14283d65111703a275c5735fe33762efafa179cb0f48e"}
Sep 11 12:27:18.723585 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.723557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"9c413e1a7591c28af1b29a70360e8f47b4da723d27a854eaa116521b9783100b"}
Sep 11 12:27:18.723694 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.723611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"51443fcd9973a4d9ba36de7fa1c6854082fe975c2a9641fefdcfa985bafbf8d0"}
Sep 11 12:27:18.725109 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.725089    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="ddc706c79e3535951416ca7eb52755e82b71ef36cc06b48b8e86b9f50051cd64" exitCode=0
Sep 11 12:27:18.725190 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.725165    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"ddc706c79e3535951416ca7eb52755e82b71ef36cc06b48b8e86b9f50051cd64"}
Sep 11 12:27:18.727019 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.726969    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t48d7" event={"ID":"8dd90914-8e03-4ea0-bfe8-bd7a31341a37","Type":"ContainerStarted","Data":"f4e77cf7546214f850b8fe023a9040f508aab6cc02566d4a72a965283f79e9a6"}
Sep 11 12:27:18.728548 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.728524    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn" event={"ID":"e49f375d-02d0-479f-8a69-03c9e7ebc8df","Type":"ContainerStarted","Data":"08e28c1144255754b9b3b3d93b104a021b044d66e99d14b876a353e226869bcf"}
Sep 11 12:27:18.729791 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.729760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tx7bj" event={"ID":"ebfa985d-6c0e-4bea-be94-078f3bef8d04","Type":"ContainerStarted","Data":"2ecd189c05204e7dceba811a4006d2fb0c6c9f8cc69e6a3d1ce16e97e7b4593a"}
Sep 11 12:27:18.734610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.734563    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-247.ec2.internal" podStartSLOduration=19.73455337 podStartE2EDuration="19.73455337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.734463535 +0000 UTC m=+21.576214661" watchObservedRunningTime="2026-09-11 12:27:18.73455337 +0000 UTC m=+21.576304496"
Sep 11 12:27:18.734974 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.734949    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-247.ec2.internal" podStartSLOduration=19.734940923 podStartE2EDuration="19.734940923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:01.696947883 +0000 UTC m=+4.538699009" watchObservedRunningTime="2026-09-11 12:27:18.734940923 +0000 UTC m=+21.576692074"
Sep 11 12:27:18.753261 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.753223    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5sw85" podStartSLOduration=2.494733263 podStartE2EDuration="20.753209996s" totalImagesPullingTime="18.258476733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.752810575 +0000 UTC m=+21.594561726" watchObservedRunningTime="2026-09-11 12:27:18.753209996 +0000 UTC m=+21.594961122"
Sep 11 12:27:18.774579 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.774524    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9hjxt" podStartSLOduration=2.438504377 podStartE2EDuration="20.774505885s" totalImagesPullingTime="18.336001508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.773731807 +0000 UTC m=+21.615482963" watchObservedRunningTime="2026-09-11 12:27:18.774505885 +0000 UTC m=+21.616257013"
Sep 11 12:27:18.822766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.822719    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tx7bj" podStartSLOduration=3.561271069 podStartE2EDuration="21.822703399s" totalImagesPullingTime="18.26143233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.799286673 +0000 UTC m=+21.641037810" watchObservedRunningTime="2026-09-11 12:27:18.822703399 +0000 UTC m=+21.664454525"
Sep 11 12:27:18.823130 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.823109    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t48d7" podStartSLOduration=3.568661163 podStartE2EDuration="21.823104092s" totalImagesPullingTime="18.254442929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.822545869 +0000 UTC m=+21.664297017" watchObservedRunningTime="2026-09-11 12:27:18.823104092 +0000 UTC m=+21.664855218"
Sep 11 12:27:18.877845 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:18.877802    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gt7b4" podStartSLOduration=2.603613009 podStartE2EDuration="20.877788241s" totalImagesPullingTime="18.274175232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:18.877587285 +0000 UTC m=+21.719338411" watchObservedRunningTime="2026-09-11 12:27:18.877788241 +0000 UTC m=+21.719539366"
Sep 11 12:27:19.156570 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.156530    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:27:19.157143 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.157122    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:27:19.624686 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.624619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:19.624881 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:19.624716    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:19.734648 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.734611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"c34dfd9f55578cc91bff41b8acfdaec2515524e3c3fc2bd211c756152ed6546e"}
Sep 11 12:27:19.735208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.734687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"f8dab30094d582fa16a8d8eeddfc931cecffafd7b0dc9673e32a1d7abca8f3c7"}
Sep 11 12:27:19.735208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.734723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"e8606e992ebb7165a7d0282d424109c0f2ffbe9d1826432f29ba991fd4718ba2"}
Sep 11 12:27:19.735208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.734756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"86a0f87dff1489574fe3ff0845bc57e4263d3ab75b184ed77126a9c5c6f27c48"}
Sep 11 12:27:19.735208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.734977    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:27:19.735493 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.735475    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tx7bj"
Sep 11 12:27:19.969819 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:19.969795    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 12:27:20.602361 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.602256    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T12:27:19.969813138Z","UUID":"0670e16d-9aaf-4039-b4fd-c082609daae8","Handler":null,"Name":"","Endpoint":""}
Sep 11 12:27:20.603568 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.603552    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 12:27:20.603667 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.603578    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 12:27:20.624150 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.624120    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:20.624150 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.624152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:20.624368 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:20.624221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:20.624368 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:20.624274    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:20.738829 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.738787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45btd" event={"ID":"79a19085-260b-448b-b9b6-abb6affcda26","Type":"ContainerStarted","Data":"7d07f967729a4548be8aaf6394ef7ad124626971f26deb1d595ed08de32ce744"}
Sep 11 12:27:20.741040 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.741010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn" event={"ID":"e49f375d-02d0-479f-8a69-03c9e7ebc8df","Type":"ContainerStarted","Data":"5d311b666423a985dbe86c1e721bde7d7eb635af2a1f1deb4678142ad3f15337"}
Sep 11 12:27:20.756254 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:20.756203    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-45btd" podStartSLOduration=5.497273321 podStartE2EDuration="23.756188308s" totalImagesPullingTime="18.258914987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:20.756021099 +0000 UTC m=+23.597772237" watchObservedRunningTime="2026-09-11 12:27:20.756188308 +0000 UTC m=+23.597939434"
Sep 11 12:27:21.624079 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:21.623891    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:21.624253 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:21.624182    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:21.747704 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:21.747664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"685be350b942a89b96e679b1c2c1fa7be1c8275a56574fc03cf064fb42927061"}
Sep 11 12:27:21.749960 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:21.749927    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn" event={"ID":"e49f375d-02d0-479f-8a69-03c9e7ebc8df","Type":"ContainerStarted","Data":"be5567c2f19c8df9fa57c2b78eb8a87533849de59a5e596c0d47fb8216eb489d"}
Sep 11 12:27:21.770348 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:21.770292    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnztn" podStartSLOduration=3.360155889 podStartE2EDuration="23.770279299s" totalImagesPullingTime="20.41012341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:21.770052707 +0000 UTC m=+24.611803852" watchObservedRunningTime="2026-09-11 12:27:21.770279299 +0000 UTC m=+24.612030425"
Sep 11 12:27:22.624342 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:22.624301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:22.624514 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:22.624430    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:22.624514 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:22.624484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:22.624625 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:22.624598    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:23.624146 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:23.624072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:23.624598 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:23.624185    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:24.623783 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:24.623745    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:24.623783 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:24.623775    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:24.623993 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:24.623886    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:24.623993 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:24.623974    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:25.624661 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.624490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:25.625194 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:25.624756    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:25.762214 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.762181    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" event={"ID":"c8b267d8-27ee-4dca-b5bb-7149c3aeaf11","Type":"ContainerStarted","Data":"e08e91e7ebdc57bf80597dd344afcfabb0fc2829ccaac3cbec5ff1b22727cb2b"}
Sep 11 12:27:25.762472 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.762438    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:25.764030 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.764002    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="041479ba9770b083e7e4093602ce16f31a96807832d7d09f8b686a39a668d8d8" exitCode=0
Sep 11 12:27:25.764105 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.764052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"041479ba9770b083e7e4093602ce16f31a96807832d7d09f8b686a39a668d8d8"}
Sep 11 12:27:25.777783 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.777764    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:25.797183 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:25.797147    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98" podStartSLOduration=9.467634917 podStartE2EDuration="27.797135858s" totalImagesPullingTime="18.329500941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:25.79673824 +0000 UTC m=+28.638489365" watchObservedRunningTime="2026-09-11 12:27:25.797135858 +0000 UTC m=+28.638886984"
Sep 11 12:27:26.623811 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.623788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:26.623907 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:26.623890    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:26.623962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.623944    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:26.624045 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:26.624030    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:26.768187 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.768109    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="d52d067122fe89d5a6e827a1b67f39b2d28835ae19478ec4caf5b0c334087ec8" exitCode=0
Sep 11 12:27:26.768913 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.768195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"d52d067122fe89d5a6e827a1b67f39b2d28835ae19478ec4caf5b0c334087ec8"}
Sep 11 12:27:26.768913 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.768617    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:26.768913 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.768730    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:26.784031 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.783871    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:27:26.833009 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.832981    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n94wh"]
Sep 11 12:27:26.833122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.833073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:26.833193 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:26.833172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:26.836800 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.836779    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfbtn"]
Sep 11 12:27:26.836905 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.836880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:26.836988 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:26.836970    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:26.837470 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.837451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mzvww"]
Sep 11 12:27:26.837557 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:26.837537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:26.837628 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:26.837611    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:27.774436 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:27.772914    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="834c3507d40dc1755addf66dee9f5d3cb56d24618a3d7509faa6c9c976c09cc8" exitCode=0
Sep 11 12:27:27.774880 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:27.774463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"834c3507d40dc1755addf66dee9f5d3cb56d24618a3d7509faa6c9c976c09cc8"}
Sep 11 12:27:28.624395 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:28.624364    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:28.624566 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:28.624400    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:28.624566 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:28.624406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:28.624566 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:28.624487    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:28.624723 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:28.624575    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:28.624723 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:28.624650    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:30.624862 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:30.624369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:30.624862 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:30.624535    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n94wh" podUID="04d375a1-a005-4516-8b95-1aa2c38b300a"
Sep 11 12:27:30.625388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:30.624966    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:30.625388 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:30.625200    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfbtn" podUID="08825d8c-0fb4-41a5-9624-0d87e65963a4"
Sep 11 12:27:30.625388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:30.625266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:30.625537 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:30.625445    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzvww" podUID="74dbc65a-330f-406e-9f26-34cd5e77565d"
Sep 11 12:27:31.304909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.304825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:31.305108 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.304986    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:31.305108 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.305060    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs podName:08825d8c-0fb4-41a5-9624-0d87e65963a4 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:03.305038659 +0000 UTC m=+66.146789775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs") pod "network-metrics-daemon-mfbtn" (UID: "08825d8c-0fb4-41a5-9624-0d87e65963a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:27:31.405715 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.405679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:31.405882 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.405760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:31.405882 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405820    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:31.405882 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405866    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:27:31.406004 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret podName:74dbc65a-330f-406e-9f26-34cd5e77565d nodeName:}" failed. No retries permitted until 2026-09-11 12:28:03.405866993 +0000 UTC m=+66.247618111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret") pod "global-pull-secret-syncer-mzvww" (UID: "74dbc65a-330f-406e-9f26-34cd5e77565d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:27:31.406004 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405892    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:27:31.406004 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405910    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rm7ng for pod openshift-network-diagnostics/network-check-target-n94wh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:31.406004 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.405934    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng podName:04d375a1-a005-4516-8b95-1aa2c38b300a nodeName:}" failed. No retries permitted until 2026-09-11 12:28:03.405927674 +0000 UTC m=+66.247678777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rm7ng" (UniqueName: "kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng") pod "network-check-target-n94wh" (UID: "04d375a1-a005-4516-8b95-1aa2c38b300a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:27:31.467184 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.467154    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-247.ec2.internal" event="NodeReady"
Sep 11 12:27:31.467353 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.467313    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 12:27:31.524042 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.524006    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wsqbf"]
Sep 11 12:27:31.554686 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.554657    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dnsxs"]
Sep 11 12:27:31.554840 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.554776    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.557577 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.557515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 12:27:31.557577 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.557542    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 12:27:31.557820 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.557806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-48qpm\""
Sep 11 12:27:31.575139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.575116    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnsxs"]
Sep 11 12:27:31.575238 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.575155    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wsqbf"]
Sep 11 12:27:31.575289 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.575252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.577972 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.577949    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 12:27:31.578188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.578166    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 12:27:31.578362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.578340    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 12:27:31.578438 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.578362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q7zjm\""
Sep 11 12:27:31.708265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e364c8eb-842d-4bf4-98b2-5db29632f4af-config-volume\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.708705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hgh\" (UniqueName: \"kubernetes.io/projected/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-kube-api-access-78hgh\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.708705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e364c8eb-842d-4bf4-98b2-5db29632f4af-tmp-dir\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.708705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdxh\" (UniqueName: \"kubernetes.io/projected/e364c8eb-842d-4bf4-98b2-5db29632f4af-kube-api-access-7wdxh\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.708705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-cert\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.708705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.708416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809056 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.808977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hgh\" (UniqueName: \"kubernetes.io/projected/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-kube-api-access-78hgh\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.809056 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e364c8eb-842d-4bf4-98b2-5db29632f4af-tmp-dir\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdxh\" (UniqueName: \"kubernetes.io/projected/e364c8eb-842d-4bf4-98b2-5db29632f4af-kube-api-access-7wdxh\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-cert\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.809284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809232    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809460 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e364c8eb-842d-4bf4-98b2-5db29632f4af-config-volume\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809460 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.809386    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:27:31.809555 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:31.809482    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls podName:e364c8eb-842d-4bf4-98b2-5db29632f4af nodeName:}" failed. No retries permitted until 2026-09-11 12:27:32.309461134 +0000 UTC m=+35.151212253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls") pod "dns-default-wsqbf" (UID: "e364c8eb-842d-4bf4-98b2-5db29632f4af") : secret "dns-default-metrics-tls" not found
Sep 11 12:27:31.809555 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e364c8eb-842d-4bf4-98b2-5db29632f4af-tmp-dir\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.809904 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.809883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e364c8eb-842d-4bf4-98b2-5db29632f4af-config-volume\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.814519 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.814498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-cert\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.818982 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.818953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hgh\" (UniqueName: \"kubernetes.io/projected/ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12-kube-api-access-78hgh\") pod \"ingress-canary-dnsxs\" (UID: \"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12\") " pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:31.819612 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.819568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdxh\" (UniqueName: \"kubernetes.io/projected/e364c8eb-842d-4bf4-98b2-5db29632f4af-kube-api-access-7wdxh\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:31.884463 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:31.884439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnsxs"
Sep 11 12:27:32.311979 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.311946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:32.312246 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:32.312122    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:27:32.312246 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:32.312204    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls podName:e364c8eb-842d-4bf4-98b2-5db29632f4af nodeName:}" failed. No retries permitted until 2026-09-11 12:27:33.312182115 +0000 UTC m=+36.153933223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls") pod "dns-default-wsqbf" (UID: "e364c8eb-842d-4bf4-98b2-5db29632f4af") : secret "dns-default-metrics-tls" not found
Sep 11 12:27:32.623988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.623826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:27:32.624114 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.623840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:27:32.624246 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.623849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:27:32.631977 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.631937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:27:32.631977 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.631960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:27:32.632126 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.632083    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:27:32.633182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.633155    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6d4p\""
Sep 11 12:27:32.633182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.633162    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmfdf\""
Sep 11 12:27:32.633281 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.633169    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:27:32.817090 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:32.815437    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnsxs"]
Sep 11 12:27:32.821249 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:32.821229    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcfa61d_a48d_4a8f_b176_56b9fe2d1d12.slice/crio-50385da5ccb67bb6bbd3cff5712550ae1787ebb6755c013df925e32e8f1aec5d WatchSource:0}: Error finding container 50385da5ccb67bb6bbd3cff5712550ae1787ebb6755c013df925e32e8f1aec5d: Status 404 returned error can't find the container with id 50385da5ccb67bb6bbd3cff5712550ae1787ebb6755c013df925e32e8f1aec5d
Sep 11 12:27:33.320049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:33.320016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:33.320222 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:33.320135    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:27:33.320222 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:33.320188    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls podName:e364c8eb-842d-4bf4-98b2-5db29632f4af nodeName:}" failed. No retries permitted until 2026-09-11 12:27:35.320175219 +0000 UTC m=+38.161926328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls") pod "dns-default-wsqbf" (UID: "e364c8eb-842d-4bf4-98b2-5db29632f4af") : secret "dns-default-metrics-tls" not found
Sep 11 12:27:33.789405 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:33.789371    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="633d73283b8c9ba8071bf8a399469aa4a1dc3d01135ae017196305f1da125905" exitCode=0
Sep 11 12:27:33.789573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:33.789459    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"633d73283b8c9ba8071bf8a399469aa4a1dc3d01135ae017196305f1da125905"}
Sep 11 12:27:33.790887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:33.790863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnsxs" event={"ID":"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12","Type":"ContainerStarted","Data":"50385da5ccb67bb6bbd3cff5712550ae1787ebb6755c013df925e32e8f1aec5d"}
Sep 11 12:27:34.795590 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:34.795559    2575 generic.go:356] "Generic (PLEG): container finished" podID="50f30488-bba7-4cb1-83ec-e9e58eb6000e" containerID="e5e43f6b1a8cff1487aa13d061ef0c2a14309f7df1fd98436fba3d088f21b1f1" exitCode=0
Sep 11 12:27:34.796184 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:34.795610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerDied","Data":"e5e43f6b1a8cff1487aa13d061ef0c2a14309f7df1fd98436fba3d088f21b1f1"}
Sep 11 12:27:35.334431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:35.334394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:35.334607 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:35.334564    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:27:35.334677 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:35.334638    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls podName:e364c8eb-842d-4bf4-98b2-5db29632f4af nodeName:}" failed. No retries permitted until 2026-09-11 12:27:39.334621794 +0000 UTC m=+42.176372897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls") pod "dns-default-wsqbf" (UID: "e364c8eb-842d-4bf4-98b2-5db29632f4af") : secret "dns-default-metrics-tls" not found
Sep 11 12:27:35.800981 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:35.800956    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" event={"ID":"50f30488-bba7-4cb1-83ec-e9e58eb6000e","Type":"ContainerStarted","Data":"744c2cfe18157be0016009d82fa8edf4b2fc18b9bbb967a57146fdf693ed91ab"}
Sep 11 12:27:35.847533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:35.847486    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9b6k9" podStartSLOduration=6.783623416 podStartE2EDuration="37.847470706s" totalImagesPullingTime="31.06384729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:35.845513008 +0000 UTC m=+38.687264153" watchObservedRunningTime="2026-09-11 12:27:35.847470706 +0000 UTC m=+38.689221832"
Sep 11 12:27:36.272886 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.272707    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"]
Sep 11 12:27:36.287770 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.287752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"
Sep 11 12:27:36.289494 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.289463    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"]
Sep 11 12:27:36.290604 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.290573    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 12:27:36.290710 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.290659    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n96mt\""
Sep 11 12:27:36.290784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.290659    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 12:27:36.331601 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.331574    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jdxzw"]
Sep 11 12:27:36.355591 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.355567    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdxzw"]
Sep 11 12:27:36.355704 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.355678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.358587 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.358568    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 12:27:36.358587 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.358582    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 12:27:36.358774 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.358593    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 12:27:36.358774 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.358596    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9m2p7\""
Sep 11 12:27:36.358774 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.358596    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 12:27:36.442879 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.442840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cks75\" (UniqueName: \"kubernetes.io/projected/e9efa306-b755-430c-a151-1161d9292d43-kube-api-access-cks75\") pod \"migrator-8b4d697c5-n47v2\" (UID: \"e9efa306-b755-430c-a151-1161d9292d43\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"
Sep 11 12:27:36.543867 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cks75\" (UniqueName: \"kubernetes.io/projected/e9efa306-b755-430c-a151-1161d9292d43-kube-api-access-cks75\") pod \"migrator-8b4d697c5-n47v2\" (UID: \"e9efa306-b755-430c-a151-1161d9292d43\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"
Sep 11 12:27:36.543867 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56774c0e-65c3-4ca1-ac11-83db1757dc0d-crio-socket\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.543867 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.543867 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.544088 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543897    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfdb\" (UniqueName: \"kubernetes.io/projected/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-api-access-jwfdb\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.544088 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.543987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56774c0e-65c3-4ca1-ac11-83db1757dc0d-data-volume\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.554501 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.554472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cks75\" (UniqueName: \"kubernetes.io/projected/e9efa306-b755-430c-a151-1161d9292d43-kube-api-access-cks75\") pod \"migrator-8b4d697c5-n47v2\" (UID: \"e9efa306-b755-430c-a151-1161d9292d43\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"
Sep 11 12:27:36.597162 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.597135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"
Sep 11 12:27:36.644947 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.644920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56774c0e-65c3-4ca1-ac11-83db1757dc0d-data-volume\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.644969    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56774c0e-65c3-4ca1-ac11-83db1757dc0d-crio-socket\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.644997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645189 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:36.645062    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:27:36.645189 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:36.645125    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:37.145108276 +0000 UTC m=+39.986859402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : configmap "kube-rbac-proxy" not found
Sep 11 12:27:36.645189 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.645123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645189 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.645166    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/56774c0e-65c3-4ca1-ac11-83db1757dc0d-crio-socket\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645421 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.645199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfdb\" (UniqueName: \"kubernetes.io/projected/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-api-access-jwfdb\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.645421 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:36.645308    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:36.645421 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:36.645387    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:37.145370036 +0000 UTC m=+39.987121144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:36.645421 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.645305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/56774c0e-65c3-4ca1-ac11-83db1757dc0d-data-volume\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.656674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.656647    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfdb\" (UniqueName: \"kubernetes.io/projected/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-api-access-jwfdb\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:36.805029 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.804928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnsxs" event={"ID":"ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12","Type":"ContainerStarted","Data":"c4c4760f87740d6abd504deb271553a52afa8e39979f99545bdbb902ce44e5b0"}
Sep 11 12:27:36.823319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.823251    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dnsxs" podStartSLOduration=2.851717409 podStartE2EDuration="5.823232048s" totalImagesPullingTime="2.971514639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:36.822941127 +0000 UTC m=+39.664692277" watchObservedRunningTime="2026-09-11 12:27:36.823232048 +0000 UTC m=+39.664983175"
Sep 11 12:27:36.891754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:36.891719    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2"]
Sep 11 12:27:36.895359 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:36.895299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9efa306_b755_430c_a151_1161d9292d43.slice/crio-fc13c623a58117ea3178becd26dfead8b9c32dd9bc9d7f1630707e7306a5851d WatchSource:0}: Error finding container fc13c623a58117ea3178becd26dfead8b9c32dd9bc9d7f1630707e7306a5851d: Status 404 returned error can't find the container with id fc13c623a58117ea3178becd26dfead8b9c32dd9bc9d7f1630707e7306a5851d
Sep 11 12:27:37.148341 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.148254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:37.148341 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.148287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:37.148506 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:37.148389    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:27:37.148506 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:37.148400    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:37.148506 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:37.148453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:38.148435581 +0000 UTC m=+40.990186702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:37.148506 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:37.148474    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:38.148463228 +0000 UTC m=+40.990214346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : configmap "kube-rbac-proxy" not found
Sep 11 12:27:37.611961 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.611927    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6t7pd"]
Sep 11 12:27:37.631679 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.631654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6t7pd"]
Sep 11 12:27:37.631821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.631787    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.638515 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.638490    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 12:27:37.640478 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.640455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 12:27:37.640971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.640937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 12:27:37.641340 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.641308    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 12:27:37.641454 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.641403    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5g262\""
Sep 11 12:27:37.753475 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.753441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fe12042-7d27-44a1-baae-f044458c11ad-signing-key\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.753643 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.753539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkkf\" (UniqueName: \"kubernetes.io/projected/0fe12042-7d27-44a1-baae-f044458c11ad-kube-api-access-gzkkf\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.753643 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.753607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fe12042-7d27-44a1-baae-f044458c11ad-signing-cabundle\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.809152 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.809116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2" event={"ID":"e9efa306-b755-430c-a151-1161d9292d43","Type":"ContainerStarted","Data":"fc13c623a58117ea3178becd26dfead8b9c32dd9bc9d7f1630707e7306a5851d"}
Sep 11 12:27:37.854960 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.854928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkkf\" (UniqueName: \"kubernetes.io/projected/0fe12042-7d27-44a1-baae-f044458c11ad-kube-api-access-gzkkf\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.855093 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.854999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fe12042-7d27-44a1-baae-f044458c11ad-signing-cabundle\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.855093 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.855036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fe12042-7d27-44a1-baae-f044458c11ad-signing-key\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.855654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.855632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fe12042-7d27-44a1-baae-f044458c11ad-signing-cabundle\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.857846 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.857824    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fe12042-7d27-44a1-baae-f044458c11ad-signing-key\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.867833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.867774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkkf\" (UniqueName: \"kubernetes.io/projected/0fe12042-7d27-44a1-baae-f044458c11ad-kube-api-access-gzkkf\") pod \"service-ca-568869fdc5-6t7pd\" (UID: \"0fe12042-7d27-44a1-baae-f044458c11ad\") " pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:37.947529 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:37.947488    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6t7pd"
Sep 11 12:27:38.043821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.043799    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5sw85_de85618e-7d30-44b9-87f0-dda082e4718e/dns-node-resolver/0.log"
Sep 11 12:27:38.157476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.157396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:38.157476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.157436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:38.157668 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:38.157541    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:27:38.157668 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:38.157565    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:38.157668 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:38.157617    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:40.157597842 +0000 UTC m=+42.999348968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : configmap "kube-rbac-proxy" not found
Sep 11 12:27:38.157668 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:38.157636    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:40.15762726 +0000 UTC m=+42.999378369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:38.249276 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.249244    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6t7pd"]
Sep 11 12:27:38.322262 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:38.322221    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe12042_7d27_44a1_baae_f044458c11ad.slice/crio-c0667e3b4803b2520c8eaadb49601f727a19aabca771e6810833bcc2dca8c1e6 WatchSource:0}: Error finding container c0667e3b4803b2520c8eaadb49601f727a19aabca771e6810833bcc2dca8c1e6: Status 404 returned error can't find the container with id c0667e3b4803b2520c8eaadb49601f727a19aabca771e6810833bcc2dca8c1e6
Sep 11 12:27:38.813164 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.813131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2" event={"ID":"e9efa306-b755-430c-a151-1161d9292d43","Type":"ContainerStarted","Data":"898acab19f98a3ff36b58f2e381f008d29b3c54ebb5efe934945b293a48af78e"}
Sep 11 12:27:38.814694 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:38.814671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6t7pd" event={"ID":"0fe12042-7d27-44a1-baae-f044458c11ad","Type":"ContainerStarted","Data":"c0667e3b4803b2520c8eaadb49601f727a19aabca771e6810833bcc2dca8c1e6"}
Sep 11 12:27:39.367804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:39.367764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:39.367992 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:39.367896    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:27:39.367992 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:39.367967    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls podName:e364c8eb-842d-4bf4-98b2-5db29632f4af nodeName:}" failed. No retries permitted until 2026-09-11 12:27:47.36794769 +0000 UTC m=+50.209698799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls") pod "dns-default-wsqbf" (UID: "e364c8eb-842d-4bf4-98b2-5db29632f4af") : secret "dns-default-metrics-tls" not found
Sep 11 12:27:39.445204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:39.445173    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t48d7_8dd90914-8e03-4ea0-bfe8-bd7a31341a37/node-ca/0.log"
Sep 11 12:27:39.818541 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:39.818501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2" event={"ID":"e9efa306-b755-430c-a151-1161d9292d43","Type":"ContainerStarted","Data":"a7f4ecb360c101adc640349215f45e235cfba0378a2cd5c5b5fb2fd866f4f37b"}
Sep 11 12:27:39.850187 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:39.850140    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n47v2" podStartSLOduration=2.052884424 podStartE2EDuration="3.850125496s" totalImagesPullingTime="1.797241072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:39.848865881 +0000 UTC m=+42.690617007" watchObservedRunningTime="2026-09-11 12:27:39.850125496 +0000 UTC m=+42.691876613"
Sep 11 12:27:40.048261 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:40.048219    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnsxs_ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12/serve-healthcheck-canary/0.log"
Sep 11 12:27:40.173099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:40.173009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:40.173099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:40.173077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:40.173367 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:40.173121    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:27:40.173367 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:40.173200    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:44.173178883 +0000 UTC m=+47.014929991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : configmap "kube-rbac-proxy" not found
Sep 11 12:27:40.173367 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:40.173218    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:40.173367 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:40.173267    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:44.173252004 +0000 UTC m=+47.015003111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:41.824650 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:41.824615    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6t7pd" event={"ID":"0fe12042-7d27-44a1-baae-f044458c11ad","Type":"ContainerStarted","Data":"0d26fff744f750850907d9d27679c42bf8cac7b1c53306c4c1ca43aff36f3796"}
Sep 11 12:27:44.205016 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:44.204976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:44.205016 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:44.205017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:44.205482 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:44.205116    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:44.205482 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:44.205151    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:27:44.205482 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:44.205178    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:52.205159708 +0000 UTC m=+55.046910817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:44.205482 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:44.205208    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:27:52.20519706 +0000 UTC m=+55.046948173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : configmap "kube-rbac-proxy" not found
Sep 11 12:27:47.430821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.430776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:47.433062 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.433040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e364c8eb-842d-4bf4-98b2-5db29632f4af-metrics-tls\") pod \"dns-default-wsqbf\" (UID: \"e364c8eb-842d-4bf4-98b2-5db29632f4af\") " pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:47.465196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.465162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:47.766360 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.766288    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6t7pd" podStartSLOduration=8.293240948 podStartE2EDuration="10.766270505s" totalImagesPullingTime="2.473029557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:41.857771355 +0000 UTC m=+44.699522506" watchObservedRunningTime="2026-09-11 12:27:47.766270505 +0000 UTC m=+50.608021654"
Sep 11 12:27:47.766823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.766805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wsqbf"]
Sep 11 12:27:47.769760 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:27:47.769733    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode364c8eb_842d_4bf4_98b2_5db29632f4af.slice/crio-b21a1f2ae5958f2a19af9ac139bc4020a1c1f876d1ef3d97297e0c8eb44cc8c3 WatchSource:0}: Error finding container b21a1f2ae5958f2a19af9ac139bc4020a1c1f876d1ef3d97297e0c8eb44cc8c3: Status 404 returned error can't find the container with id b21a1f2ae5958f2a19af9ac139bc4020a1c1f876d1ef3d97297e0c8eb44cc8c3
Sep 11 12:27:47.841935 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:47.841898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsqbf" event={"ID":"e364c8eb-842d-4bf4-98b2-5db29632f4af","Type":"ContainerStarted","Data":"b21a1f2ae5958f2a19af9ac139bc4020a1c1f876d1ef3d97297e0c8eb44cc8c3"}
Sep 11 12:27:50.855658 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:50.855608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsqbf" event={"ID":"e364c8eb-842d-4bf4-98b2-5db29632f4af","Type":"ContainerStarted","Data":"aee9fc20f334c0ecaa54dd493918168bb2f1865ddfda4dc2138d1f48d8841a13"}
Sep 11 12:27:50.856101 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:50.855706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsqbf" event={"ID":"e364c8eb-842d-4bf4-98b2-5db29632f4af","Type":"ContainerStarted","Data":"646512f1910827e62f981c82d44d7a5fdbfd3c0ab5c09bec5b0948b70aa2683a"}
Sep 11 12:27:50.856101 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:50.855762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:27:50.879791 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:50.879733    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wsqbf" podStartSLOduration=17.676820718 podStartE2EDuration="19.879713715s" totalImagesPullingTime="2.202892997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:50.878486027 +0000 UTC m=+53.720237157" watchObservedRunningTime="2026-09-11 12:27:50.879713715 +0000 UTC m=+53.721464833"
Sep 11 12:27:52.267433 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:52.267386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:52.267433 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:52.267429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:52.268069 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:52.267614    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:52.268069 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:27:52.267702    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls podName:56774c0e-65c3-4ca1-ac11-83db1757dc0d nodeName:}" failed. No retries permitted until 2026-09-11 12:28:08.267685215 +0000 UTC m=+71.109436343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jdxzw" (UID: "56774c0e-65c3-4ca1-ac11-83db1757dc0d") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:27:52.268069 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:52.268050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/56774c0e-65c3-4ca1-ac11-83db1757dc0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:27:58.786783 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:27:58.786745    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqn98"
Sep 11 12:28:00.860708 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:00.860676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wsqbf"
Sep 11 12:28:02.916039 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.916001    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"]
Sep 11 12:28:02.919945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.919922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:02.921279 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.921254    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"]
Sep 11 12:28:02.922899 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.922879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:02.928579 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.928564    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 12:28:02.941612 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.941588    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 12:28:02.942455 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.942437    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 12:28:02.942575 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.942557    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 12:28:02.942677 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.942659    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sfvjr\""
Sep 11 12:28:02.962646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.962621    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 12:28:02.967726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.967708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"]
Sep 11 12:28:02.976297 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:02.976279    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"]
Sep 11 12:28:03.043818 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.043794    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnfh\" (UniqueName: \"kubernetes.io/projected/320bf552-31d5-4115-8fce-2a313d8c10c5-kube-api-access-6rnfh\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.043943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.043824    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c5ecab-720c-41e3-923c-dfe4d04f483e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.043943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.043846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/320bf552-31d5-4115-8fce-2a313d8c10c5-tmp\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.043943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.043925    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxd8k\" (UniqueName: \"kubernetes.io/projected/30c5ecab-720c-41e3-923c-dfe4d04f483e-kube-api-access-xxd8k\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.044046 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.043954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/320bf552-31d5-4115-8fce-2a313d8c10c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.144478 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxd8k\" (UniqueName: \"kubernetes.io/projected/30c5ecab-720c-41e3-923c-dfe4d04f483e-kube-api-access-xxd8k\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.144478 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/320bf552-31d5-4115-8fce-2a313d8c10c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.144652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnfh\" (UniqueName: \"kubernetes.io/projected/320bf552-31d5-4115-8fce-2a313d8c10c5-kube-api-access-6rnfh\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.144652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c5ecab-720c-41e3-923c-dfe4d04f483e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.144652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/320bf552-31d5-4115-8fce-2a313d8c10c5-tmp\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.144991 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.144967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/320bf552-31d5-4115-8fce-2a313d8c10c5-tmp\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.147025 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.147003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/320bf552-31d5-4115-8fce-2a313d8c10c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.147133 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.147107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c5ecab-720c-41e3-923c-dfe4d04f483e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.162426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.162405    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-574c6bf895-twjkf"]
Sep 11 12:28:03.166696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.166657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.170248 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.170229    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 12:28:03.170389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.170369    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7zqm8\""
Sep 11 12:28:03.171053 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.171039    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 12:28:03.171975 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.171954    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 12:28:03.183827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.183802    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 12:28:03.184664 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.184643    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnfh\" (UniqueName: \"kubernetes.io/projected/320bf552-31d5-4115-8fce-2a313d8c10c5-kube-api-access-6rnfh\") pod \"klusterlet-addon-workmgr-6864d9f456-fqx5f\" (UID: \"320bf552-31d5-4115-8fce-2a313d8c10c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.185889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.185871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxd8k\" (UniqueName: \"kubernetes.io/projected/30c5ecab-720c-41e3-923c-dfe4d04f483e-kube-api-access-xxd8k\") pod \"managed-serviceaccount-addon-agent-584df6b94d-j4g7h\" (UID: \"30c5ecab-720c-41e3-923c-dfe4d04f483e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.203957 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.203937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574c6bf895-twjkf"]
Sep 11 12:28:03.240357 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.240320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"
Sep 11 12:28:03.245632 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:03.245722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56l4h\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-kube-api-access-56l4h\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245765 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-tls\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245799 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-trusted-ca\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245857 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-certificates\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245891 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-ca-trust-extracted\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245935 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-image-registry-private-configuration\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245935 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-installation-pull-secrets\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.245999 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.245948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-bound-sa-token\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346445 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-trusted-ca\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346531 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:28:03.346531 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-certificates\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346609 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-ca-trust-extracted\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346609 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-image-registry-private-configuration\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346669 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-installation-pull-secrets\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346669 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-bound-sa-token\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346732 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56l4h\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-kube-api-access-56l4h\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.346732 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.346703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-tls\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.347442 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.347417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-trusted-ca\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.347559 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.347417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-certificates\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.347559 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.347492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-ca-trust-extracted\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.349460 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.349437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-image-registry-private-configuration\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.349848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.349826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-installation-pull-secrets\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.350386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.350306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-registry-tls\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.352012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.351995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:28:03.359000 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.358983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08825d8c-0fb4-41a5-9624-0d87e65963a4-metrics-certs\") pod \"network-metrics-daemon-mfbtn\" (UID: \"08825d8c-0fb4-41a5-9624-0d87e65963a4\") " pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:28:03.361210 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.361190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-bound-sa-token\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.368575 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.368555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56l4h\" (UniqueName: \"kubernetes.io/projected/6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a-kube-api-access-56l4h\") pod \"image-registry-574c6bf895-twjkf\" (UID: \"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a\") " pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.447576 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.447546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:28:03.447721 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.447587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:28:03.452557 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.452536    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:28:03.452720 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.452701    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:28:03.460146 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.460125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74dbc65a-330f-406e-9f26-34cd5e77565d-original-pull-secret\") pod \"global-pull-secret-syncer-mzvww\" (UID: \"74dbc65a-330f-406e-9f26-34cd5e77565d\") " pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:28:03.461054 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.461039    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:28:03.470168 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.470147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7ng\" (UniqueName: \"kubernetes.io/projected/04d375a1-a005-4516-8b95-1aa2c38b300a-kube-api-access-rm7ng\") pod \"network-check-target-n94wh\" (UID: \"04d375a1-a005-4516-8b95-1aa2c38b300a\") " pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:28:03.475418 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.475403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:03.536012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.535988    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmfdf\""
Sep 11 12:28:03.538989 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.538957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzvww"
Sep 11 12:28:03.540548 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.540525    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h"]
Sep 11 12:28:03.543875 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.543853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfbtn"
Sep 11 12:28:03.545465 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.545437    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c5ecab_720c_41e3_923c_dfe4d04f483e.slice/crio-3abe7edea9b595b29f67cd443d7398cf27c8a122210bb0b7ed2670fc715eb404 WatchSource:0}: Error finding container 3abe7edea9b595b29f67cd443d7398cf27c8a122210bb0b7ed2670fc715eb404: Status 404 returned error can't find the container with id 3abe7edea9b595b29f67cd443d7398cf27c8a122210bb0b7ed2670fc715eb404
Sep 11 12:28:03.548436 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.548417    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6d4p\""
Sep 11 12:28:03.554156 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.554035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:28:03.577896 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.577870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"]
Sep 11 12:28:03.580492 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.580468    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320bf552_31d5_4115_8fce_2a313d8c10c5.slice/crio-0f8a03af0de2b1b09a898871ff9fe74ad8503d7b590e8142892718ce7bf4f7a7 WatchSource:0}: Error finding container 0f8a03af0de2b1b09a898871ff9fe74ad8503d7b590e8142892718ce7bf4f7a7: Status 404 returned error can't find the container with id 0f8a03af0de2b1b09a898871ff9fe74ad8503d7b590e8142892718ce7bf4f7a7
Sep 11 12:28:03.792514 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.792401    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574c6bf895-twjkf"]
Sep 11 12:28:03.794782 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.794757    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4dcdc3_844b_4c4b_a07f_0d26be59bf6a.slice/crio-0334db19b085438d8988c9e07c226c3017c67d88d5f6923ec9f8dc824bc6f757 WatchSource:0}: Error finding container 0334db19b085438d8988c9e07c226c3017c67d88d5f6923ec9f8dc824bc6f757: Status 404 returned error can't find the container with id 0334db19b085438d8988c9e07c226c3017c67d88d5f6923ec9f8dc824bc6f757
Sep 11 12:28:03.836746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.836717    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mzvww"]
Sep 11 12:28:03.839991 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.839968    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dbc65a_330f_406e_9f26_34cd5e77565d.slice/crio-1a7c12fa7a8dc50859778134f75c61c093755289fc6f08784b32ae64fb138094 WatchSource:0}: Error finding container 1a7c12fa7a8dc50859778134f75c61c093755289fc6f08784b32ae64fb138094: Status 404 returned error can't find the container with id 1a7c12fa7a8dc50859778134f75c61c093755289fc6f08784b32ae64fb138094
Sep 11 12:28:03.863882 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.863861    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfbtn"]
Sep 11 12:28:03.872161 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.872139    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08825d8c_0fb4_41a5_9624_0d87e65963a4.slice/crio-2a4cfa170581f89ec7c713d1f9b2a614aff25ea8ed0db0ef49b445cbdf8a38b3 WatchSource:0}: Error finding container 2a4cfa170581f89ec7c713d1f9b2a614aff25ea8ed0db0ef49b445cbdf8a38b3: Status 404 returned error can't find the container with id 2a4cfa170581f89ec7c713d1f9b2a614aff25ea8ed0db0ef49b445cbdf8a38b3
Sep 11 12:28:03.894511 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.894486    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h" event={"ID":"30c5ecab-720c-41e3-923c-dfe4d04f483e","Type":"ContainerStarted","Data":"3abe7edea9b595b29f67cd443d7398cf27c8a122210bb0b7ed2670fc715eb404"}
Sep 11 12:28:03.895602 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.895576    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f" event={"ID":"320bf552-31d5-4115-8fce-2a313d8c10c5","Type":"ContainerStarted","Data":"0f8a03af0de2b1b09a898871ff9fe74ad8503d7b590e8142892718ce7bf4f7a7"}
Sep 11 12:28:03.896642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.896624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfbtn" event={"ID":"08825d8c-0fb4-41a5-9624-0d87e65963a4","Type":"ContainerStarted","Data":"2a4cfa170581f89ec7c713d1f9b2a614aff25ea8ed0db0ef49b445cbdf8a38b3"}
Sep 11 12:28:03.897769 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.897746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mzvww" event={"ID":"74dbc65a-330f-406e-9f26-34cd5e77565d","Type":"ContainerStarted","Data":"1a7c12fa7a8dc50859778134f75c61c093755289fc6f08784b32ae64fb138094"}
Sep 11 12:28:03.899222 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.899201    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n94wh"]
Sep 11 12:28:03.899315 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:03.899252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c6bf895-twjkf" event={"ID":"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a","Type":"ContainerStarted","Data":"0334db19b085438d8988c9e07c226c3017c67d88d5f6923ec9f8dc824bc6f757"}
Sep 11 12:28:03.901432 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:03.901407    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d375a1_a005_4516_8b95_1aa2c38b300a.slice/crio-91937bc4664313c692ad873656953f3de03327a91ed517b1bc71c0a02829a156 WatchSource:0}: Error finding container 91937bc4664313c692ad873656953f3de03327a91ed517b1bc71c0a02829a156: Status 404 returned error can't find the container with id 91937bc4664313c692ad873656953f3de03327a91ed517b1bc71c0a02829a156
Sep 11 12:28:04.932454 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:04.932340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n94wh" event={"ID":"04d375a1-a005-4516-8b95-1aa2c38b300a","Type":"ContainerStarted","Data":"91937bc4664313c692ad873656953f3de03327a91ed517b1bc71c0a02829a156"}
Sep 11 12:28:04.940865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:04.940805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574c6bf895-twjkf" event={"ID":"6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a","Type":"ContainerStarted","Data":"bb8c0e58212a48a5099d901aa8df539f68897529903d6801321572fd7f01b6e5"}
Sep 11 12:28:04.941685 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:04.941632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:04.969317 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:04.968572    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-574c6bf895-twjkf" podStartSLOduration=1.968529165 podStartE2EDuration="1.968529165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:04.966983193 +0000 UTC m=+67.808734341" watchObservedRunningTime="2026-09-11 12:28:04.968529165 +0000 UTC m=+67.810280286"
Sep 11 12:28:08.286766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:08.286729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:28:08.289171 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:08.289149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/56774c0e-65c3-4ca1-ac11-83db1757dc0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdxzw\" (UID: \"56774c0e-65c3-4ca1-ac11-83db1757dc0d\") " pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:28:08.466775 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:08.466738    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9m2p7\""
Sep 11 12:28:08.474619 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:08.474595    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdxzw"
Sep 11 12:28:12.195936 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.195906    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdxzw"]
Sep 11 12:28:12.199169 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:12.199142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56774c0e_65c3_4ca1_ac11_83db1757dc0d.slice/crio-42855ba88497bfff588936316951c77ec401c9ee1abdea3553a17ffe6a39df59 WatchSource:0}: Error finding container 42855ba88497bfff588936316951c77ec401c9ee1abdea3553a17ffe6a39df59: Status 404 returned error can't find the container with id 42855ba88497bfff588936316951c77ec401c9ee1abdea3553a17ffe6a39df59
Sep 11 12:28:12.985397 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.985361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfbtn" event={"ID":"08825d8c-0fb4-41a5-9624-0d87e65963a4","Type":"ContainerStarted","Data":"536fc6b2214df0ab572bc1d661a5191598460fefed6bc8e3b4127bcec8f7d9da"}
Sep 11 12:28:12.985554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.985429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfbtn" event={"ID":"08825d8c-0fb4-41a5-9624-0d87e65963a4","Type":"ContainerStarted","Data":"bb403c69cbfe23863ddfa5346ac0ebde327ced0b3970ca17e580bca104baeb80"}
Sep 11 12:28:12.986775 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.986742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mzvww" event={"ID":"74dbc65a-330f-406e-9f26-34cd5e77565d","Type":"ContainerStarted","Data":"c99e1bddc47429964e2ade25c0a910917574d76deb79b263b68aea6af6eef4fd"}
Sep 11 12:28:12.988114 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.988088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h" event={"ID":"30c5ecab-720c-41e3-923c-dfe4d04f483e","Type":"ContainerStarted","Data":"868c200e51a6d0d46b58ad193a7154f97d3a11fac66779bb72ffc0e446979c57"}
Sep 11 12:28:12.989586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.989560    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n94wh" event={"ID":"04d375a1-a005-4516-8b95-1aa2c38b300a","Type":"ContainerStarted","Data":"35e557ba8dc4b70a5781bb99a8b85c8b264b7d3e1c9108dec2b8db11bc2fad8f"}
Sep 11 12:28:12.989676 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.989610    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:28:12.991340 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.991302    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f" event={"ID":"320bf552-31d5-4115-8fce-2a313d8c10c5","Type":"ContainerStarted","Data":"882d322b0f1f622245fe6bd59685ff5647bfa081babef1f039b58f8c378fe6b5"}
Sep 11 12:28:12.991556 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.991533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:12.993773 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.993749    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f"
Sep 11 12:28:12.997529 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.997506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdxzw" event={"ID":"56774c0e-65c3-4ca1-ac11-83db1757dc0d","Type":"ContainerStarted","Data":"c33263afc88158942e50ea2182ada639ae2ef03494279b1bcd2a30514f8fbca9"}
Sep 11 12:28:12.997610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:12.997554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdxzw" event={"ID":"56774c0e-65c3-4ca1-ac11-83db1757dc0d","Type":"ContainerStarted","Data":"42855ba88497bfff588936316951c77ec401c9ee1abdea3553a17ffe6a39df59"}
Sep 11 12:28:13.007725 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:13.007686    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mfbtn" podStartSLOduration=66.977073032 podStartE2EDuration="1m15.007674742s" totalImagesPullingTime="8.03060171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:13.006038048 +0000 UTC m=+75.847789194" watchObservedRunningTime="2026-09-11 12:28:13.007674742 +0000 UTC m=+75.849425867"
Sep 11 12:28:13.027633 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:13.027588    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n94wh" podStartSLOduration=68.016109315 podStartE2EDuration="1m16.027574849s" totalImagesPullingTime="8.011465534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:13.026748463 +0000 UTC m=+75.868499611" watchObservedRunningTime="2026-09-11 12:28:13.027574849 +0000 UTC m=+75.869325976"
Sep 11 12:28:13.054462 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:13.054403    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mzvww" podStartSLOduration=66.968864933 podStartE2EDuration="1m15.054385346s" totalImagesPullingTime="8.085520413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:13.053683145 +0000 UTC m=+75.895434291" watchObservedRunningTime="2026-09-11 12:28:13.054385346 +0000 UTC m=+75.896136472"
Sep 11 12:28:13.084808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:13.084757    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-584df6b94d-j4g7h" podStartSLOduration=2.722774625 podStartE2EDuration="11.084742523s" totalImagesPullingTime="8.361967898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:13.083080921 +0000 UTC m=+75.924832044" watchObservedRunningTime="2026-09-11 12:28:13.084742523 +0000 UTC m=+75.926493646"
Sep 11 12:28:13.111059 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:13.111018    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864d9f456-fqx5f" podStartSLOduration=2.7671195429999997 podStartE2EDuration="11.111002102s" totalImagesPullingTime="8.343882559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:13.110300429 +0000 UTC m=+75.952051573" watchObservedRunningTime="2026-09-11 12:28:13.111002102 +0000 UTC m=+75.952753234"
Sep 11 12:28:14.003657 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.003617    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdxzw" event={"ID":"56774c0e-65c3-4ca1-ac11-83db1757dc0d","Type":"ContainerStarted","Data":"85a0fc784ce10d6da9182973033a7c7373aaf5d0fcdcd151255cb3474b78a57d"}
Sep 11 12:28:14.180956 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.180905    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sd4lz"]
Sep 11 12:28:14.183598 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.183578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.186942 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.186918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 12:28:14.187264 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.187246    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 12:28:14.187264 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.187253    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pz2h4\""
Sep 11 12:28:14.187446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.187417    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 12:28:14.188481 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.188464    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 12:28:14.188569 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.188511    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 12:28:14.188569 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.188517    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 12:28:14.229550 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-root\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229550 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-wtmp\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-sys\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgpw\" (UniqueName: \"kubernetes.io/projected/d37e9227-9205-48a9-85ed-f0fd03c0d37b-kube-api-access-bjgpw\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-textfile\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.229784 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-metrics-client-ca\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.230012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.229802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.330805 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-wtmp\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.330805 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-sys\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgpw\" (UniqueName: \"kubernetes.io/projected/d37e9227-9205-48a9-85ed-f0fd03c0d37b-kube-api-access-bjgpw\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-wtmp\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-textfile\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-metrics-client-ca\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:28:14.330996    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 12:28:14.331022 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.331025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-root\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331484 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:28:14.331067    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls podName:d37e9227-9205-48a9-85ed-f0fd03c0d37b nodeName:}" failed. No retries permitted until 2026-09-11 12:28:14.831046143 +0000 UTC m=+77.672797251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls") pod "node-exporter-sd4lz" (UID: "d37e9227-9205-48a9-85ed-f0fd03c0d37b") : secret "node-exporter-tls" not found
Sep 11 12:28:14.331484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.331088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-root\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.330901    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d37e9227-9205-48a9-85ed-f0fd03c0d37b-sys\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.331723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-metrics-client-ca\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331859 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.331835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.331988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.331970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-textfile\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.333424 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.333397    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.341841 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.341804    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgpw\" (UniqueName: \"kubernetes.io/projected/d37e9227-9205-48a9-85ed-f0fd03c0d37b-kube-api-access-bjgpw\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.835865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.835825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:14.838301 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:14.838271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d37e9227-9205-48a9-85ed-f0fd03c0d37b-node-exporter-tls\") pod \"node-exporter-sd4lz\" (UID: \"d37e9227-9205-48a9-85ed-f0fd03c0d37b\") " pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:15.094194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:15.094108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sd4lz"
Sep 11 12:28:15.105462 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:15.105435    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37e9227_9205_48a9_85ed_f0fd03c0d37b.slice/crio-881e98ec6684beb268ae8878ba480aa5111011d6198521ae5797af24b0843b6a WatchSource:0}: Error finding container 881e98ec6684beb268ae8878ba480aa5111011d6198521ae5797af24b0843b6a: Status 404 returned error can't find the container with id 881e98ec6684beb268ae8878ba480aa5111011d6198521ae5797af24b0843b6a
Sep 11 12:28:16.016877 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:16.016832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdxzw" event={"ID":"56774c0e-65c3-4ca1-ac11-83db1757dc0d","Type":"ContainerStarted","Data":"d23a348300d3b0eb667ce35d0ee9639ba0c045756342503b9ade446b6dba90e0"}
Sep 11 12:28:16.018226 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:16.018197    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd4lz" event={"ID":"d37e9227-9205-48a9-85ed-f0fd03c0d37b","Type":"ContainerStarted","Data":"881e98ec6684beb268ae8878ba480aa5111011d6198521ae5797af24b0843b6a"}
Sep 11 12:28:16.038603 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:16.038540    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jdxzw" podStartSLOduration=36.682852441 podStartE2EDuration="40.038522893s" totalImagesPullingTime="3.355670452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:27:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:16.036506486 +0000 UTC m=+78.878257641" watchObservedRunningTime="2026-09-11 12:28:16.038522893 +0000 UTC m=+78.880274019"
Sep 11 12:28:17.023500 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.023462    2575 generic.go:356] "Generic (PLEG): container finished" podID="d37e9227-9205-48a9-85ed-f0fd03c0d37b" containerID="dddf0faee704446e0ec6a4b706d74dec988c8e2a407f18af256427517903a01c" exitCode=0
Sep 11 12:28:17.023999 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.023561    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd4lz" event={"ID":"d37e9227-9205-48a9-85ed-f0fd03c0d37b","Type":"ContainerDied","Data":"dddf0faee704446e0ec6a4b706d74dec988c8e2a407f18af256427517903a01c"}
Sep 11 12:28:17.332085 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.331994    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"]
Sep 11 12:28:17.334938 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.334913    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.339680 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.339658    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 12:28:17.339840 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.339799    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 12:28:17.340013 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.339987    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9et43njvvgbue\""
Sep 11 12:28:17.340087 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.339998    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 12:28:17.340924 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.340890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 12:28:17.340924 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.340919    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 12:28:17.341140 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.340958    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-btf6n\""
Sep 11 12:28:17.353911 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.353867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"]
Sep 11 12:28:17.459822 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.459822 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/909cb820-6630-4062-9827-a8c3009c0642-metrics-client-ca\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-grpc-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459945    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.459996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.460049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.460019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64vz\" (UniqueName: \"kubernetes.io/projected/909cb820-6630-4062-9827-a8c3009c0642-kube-api-access-b64vz\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.560882 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.560838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64vz\" (UniqueName: \"kubernetes.io/projected/909cb820-6630-4062-9827-a8c3009c0642-kube-api-access-b64vz\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.560891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.560919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/909cb820-6630-4062-9827-a8c3009c0642-metrics-client-ca\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.560953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.561020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-grpc-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.561066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561395 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.561094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561395 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.561120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.561916 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.561879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/909cb820-6630-4062-9827-a8c3009c0642-metrics-client-ca\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.563736 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.563704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.563875 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.563844    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.564061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.564040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.564222 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.564203    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-grpc-tls\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.564270 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.564220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.564270 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.564237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/909cb820-6630-4062-9827-a8c3009c0642-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.570419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.570395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64vz\" (UniqueName: \"kubernetes.io/projected/909cb820-6630-4062-9827-a8c3009c0642-kube-api-access-b64vz\") pod \"thanos-querier-79bbf8d7d5-xmm5m\" (UID: \"909cb820-6630-4062-9827-a8c3009c0642\") " pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.645103 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.645012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:17.936009 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:17.935977    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"]
Sep 11 12:28:17.938789 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:17.938756    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909cb820_6630_4062_9827_a8c3009c0642.slice/crio-c7d5bde49a964c4ef49bd926e03e27d0d18e1ef952741c66a4219d4ede17a8e7 WatchSource:0}: Error finding container c7d5bde49a964c4ef49bd926e03e27d0d18e1ef952741c66a4219d4ede17a8e7: Status 404 returned error can't find the container with id c7d5bde49a964c4ef49bd926e03e27d0d18e1ef952741c66a4219d4ede17a8e7
Sep 11 12:28:18.029684 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.029648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd4lz" event={"ID":"d37e9227-9205-48a9-85ed-f0fd03c0d37b","Type":"ContainerStarted","Data":"eae00c7dcbe003a53b46ee77a77b0cf27a134e391d8dbcc31c123299f3d10062"}
Sep 11 12:28:18.030102 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.029715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd4lz" event={"ID":"d37e9227-9205-48a9-85ed-f0fd03c0d37b","Type":"ContainerStarted","Data":"ffe7918407801601bd93c98cf048a8dd6440bd4490fdf79cd406f8822134cfd8"}
Sep 11 12:28:18.030865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.030836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"c7d5bde49a964c4ef49bd926e03e27d0d18e1ef952741c66a4219d4ede17a8e7"}
Sep 11 12:28:18.056997 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.056939    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sd4lz" podStartSLOduration=2.915673273 podStartE2EDuration="4.056922794s" totalImagesPullingTime="1.076991724s" totalInitContainerRuntime="64.257797ms" isStatefulPod=true podCreationTimestamp="2026-09-11 12:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:18.054197045 +0000 UTC m=+80.895948171" watchObservedRunningTime="2026-09-11 12:28:18.056922794 +0000 UTC m=+80.898673920"
Sep 11 12:28:18.530111 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.530071    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d75b47df-5q5bq"]
Sep 11 12:28:18.532865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.532841    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.536843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.536808    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 12:28:18.538269 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.538244    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 12:28:18.538421 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.538354    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8fbf4m2braeo2\""
Sep 11 12:28:18.538492 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.538445    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 12:28:18.538492 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.538471    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4ttwh\""
Sep 11 12:28:18.538604 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.538444    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 12:28:18.545732 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.545701    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d75b47df-5q5bq"]
Sep 11 12:28:18.670381 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-client-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb39a362-71b7-4626-9b88-0adc8f9544af-audit-log\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-metrics-server-audit-profiles\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-tls\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-client-certs\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.670750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.670687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lmj\" (UniqueName: \"kubernetes.io/projected/bb39a362-71b7-4626-9b88-0adc8f9544af-kube-api-access-s7lmj\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.771944 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.771900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb39a362-71b7-4626-9b88-0adc8f9544af-audit-log\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.771965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-metrics-server-audit-profiles\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-tls\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-client-certs\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lmj\" (UniqueName: \"kubernetes.io/projected/bb39a362-71b7-4626-9b88-0adc8f9544af-kube-api-access-s7lmj\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772414 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-client-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.772466 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.772407    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb39a362-71b7-4626-9b88-0adc8f9544af-audit-log\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.773135 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.773104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.773425 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.773392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb39a362-71b7-4626-9b88-0adc8f9544af-metrics-server-audit-profiles\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.775243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.775217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-client-ca-bundle\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.775724 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.775694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-tls\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.775820 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.775725    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb39a362-71b7-4626-9b88-0adc8f9544af-secret-metrics-server-client-certs\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.783177 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.783052    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lmj\" (UniqueName: \"kubernetes.io/projected/bb39a362-71b7-4626-9b88-0adc8f9544af-kube-api-access-s7lmj\") pod \"metrics-server-7d75b47df-5q5bq\" (UID: \"bb39a362-71b7-4626-9b88-0adc8f9544af\") " pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.844226 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.844189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:18.915442 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.914651    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"]
Sep 11 12:28:18.926638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.919756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:18.926638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.923450    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8v8lq\""
Sep 11 12:28:18.926638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.923710    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 12:28:18.932151 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.932092    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"]
Sep 11 12:28:18.973699 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.973655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvc9p\" (UniqueName: \"kubernetes.io/projected/5895766c-ffbb-4967-bc4d-13cc14a1c6db-kube-api-access-bvc9p\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:18.973889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:18.973712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.075231 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.075133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvc9p\" (UniqueName: \"kubernetes.io/projected/5895766c-ffbb-4967-bc4d-13cc14a1c6db-kube-api-access-bvc9p\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.075231 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.075197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.075770 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:28:19.075357    2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 12:28:19.075770 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:28:19.075437    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert podName:5895766c-ffbb-4967-bc4d-13cc14a1c6db nodeName:}" failed. No retries permitted until 2026-09-11 12:28:19.575414763 +0000 UTC m=+82.417166072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-d955z" (UID: "5895766c-ffbb-4967-bc4d-13cc14a1c6db") : secret "monitoring-plugin-cert" not found
Sep 11 12:28:19.089417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.089381    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvc9p\" (UniqueName: \"kubernetes.io/projected/5895766c-ffbb-4967-bc4d-13cc14a1c6db-kube-api-access-bvc9p\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.141083 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.141052    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d75b47df-5q5bq"]
Sep 11 12:28:19.144454 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:19.144428    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb39a362_71b7_4626_9b88_0adc8f9544af.slice/crio-e1a7c01107432166db4c89beaa552b38a8895addc15e88d291d3e1a0598ec837 WatchSource:0}: Error finding container e1a7c01107432166db4c89beaa552b38a8895addc15e88d291d3e1a0598ec837: Status 404 returned error can't find the container with id e1a7c01107432166db4c89beaa552b38a8895addc15e88d291d3e1a0598ec837
Sep 11 12:28:19.580831 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.580786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.583722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.583693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5895766c-ffbb-4967-bc4d-13cc14a1c6db-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-d955z\" (UID: \"5895766c-ffbb-4967-bc4d-13cc14a1c6db\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:19.842748 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:19.842698    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:20.044139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:20.043706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"712c2c41fbb09299d7cff40a208f2e8683c7b5e5401c92920339775fb2f2f7ae"}
Sep 11 12:28:20.044139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:20.043776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"a1ab05d341ead14681b5dd9b69b5989a83c02ba49d6ac3d08ce4838d69e91219"}
Sep 11 12:28:20.044139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:20.043805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"82355c49c6114a728ba0f5ed9e413bd5e5ad3d7650e674144efe483c5a5b675d"}
Sep 11 12:28:20.047131 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:20.047088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq" event={"ID":"bb39a362-71b7-4626-9b88-0adc8f9544af","Type":"ContainerStarted","Data":"e1a7c01107432166db4c89beaa552b38a8895addc15e88d291d3e1a0598ec837"}
Sep 11 12:28:20.131873 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:20.131730    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"]
Sep 11 12:28:20.134653 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:28:20.134623    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5895766c_ffbb_4967_bc4d_13cc14a1c6db.slice/crio-c06ab435855e7260d3ba4c4696f7bc6b23dacf69ffa6f45123ce2b13f4d13d8d WatchSource:0}: Error finding container c06ab435855e7260d3ba4c4696f7bc6b23dacf69ffa6f45123ce2b13f4d13d8d: Status 404 returned error can't find the container with id c06ab435855e7260d3ba4c4696f7bc6b23dacf69ffa6f45123ce2b13f4d13d8d
Sep 11 12:28:21.053731 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:21.053645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z" event={"ID":"5895766c-ffbb-4967-bc4d-13cc14a1c6db","Type":"ContainerStarted","Data":"c06ab435855e7260d3ba4c4696f7bc6b23dacf69ffa6f45123ce2b13f4d13d8d"}
Sep 11 12:28:21.056039 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:21.056010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq" event={"ID":"bb39a362-71b7-4626-9b88-0adc8f9544af","Type":"ContainerStarted","Data":"0350709f66f2a24a07e0cc69a492ccc01701f770d88ad6c51a2186f0abdadb3c"}
Sep 11 12:28:21.076567 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:21.076503    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq" podStartSLOduration=1.484544474 podStartE2EDuration="3.076483232s" totalImagesPullingTime="1.591938758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:21.076048463 +0000 UTC m=+83.917799610" watchObservedRunningTime="2026-09-11 12:28:21.076483232 +0000 UTC m=+83.918234360"
Sep 11 12:28:22.063010 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.062901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"bd2639f57f52adfa8968c3ad97b5e98df5a187c86b64eeabf23be81220c73b92"}
Sep 11 12:28:22.063010 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.062978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"ea8bdaa2d38fd22eec46ff6906010e514d350b6a065e938e9fe2d27fb0423f81"}
Sep 11 12:28:22.063519 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.063026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" event={"ID":"909cb820-6630-4062-9827-a8c3009c0642","Type":"ContainerStarted","Data":"f35f3086569f5cc7eb7505644e3260864622f7d2181c655ac0d60e22413688f6"}
Sep 11 12:28:22.063519 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.063067    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:22.064500 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.064476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z" event={"ID":"5895766c-ffbb-4967-bc4d-13cc14a1c6db","Type":"ContainerStarted","Data":"44536e2f6531eca3d66b6794387bda1cb4ed4ec69a84870fa8299bb99570badb"}
Sep 11 12:28:22.064629 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.064513    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:22.069204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.069181    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z"
Sep 11 12:28:22.089191 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.089133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m" podStartSLOduration=2.135079411 podStartE2EDuration="5.089105808s" totalImagesPullingTime="2.954026397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:22.087544055 +0000 UTC m=+84.929295226" watchObservedRunningTime="2026-09-11 12:28:22.089105808 +0000 UTC m=+84.930856933"
Sep 11 12:28:22.110177 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:22.110122    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-d955z" podStartSLOduration=2.538192839 podStartE2EDuration="4.110104741s" totalImagesPullingTime="1.571911902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:22.108611689 +0000 UTC m=+84.950362833" watchObservedRunningTime="2026-09-11 12:28:22.110104741 +0000 UTC m=+84.951855867"
Sep 11 12:28:26.955416 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:26.955383    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-574c6bf895-twjkf"
Sep 11 12:28:28.073375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:28.073342    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79bbf8d7d5-xmm5m"
Sep 11 12:28:38.844573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:38.844531    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:38.845149 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:38.844604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:44.006105 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:44.006071    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n94wh"
Sep 11 12:28:58.850515 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:58.850483    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:58.854252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:58.854222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d75b47df-5q5bq"
Sep 11 12:28:59.399112 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:59.399083    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d75b47df-5q5bq_bb39a362-71b7-4626-9b88-0adc8f9544af/metrics-server/0.log"
Sep 11 12:28:59.599084 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:28:59.599059    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-d955z_5895766c-ffbb-4967-bc4d-13cc14a1c6db/monitoring-plugin/0.log"
Sep 11 12:29:00.398565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:00.398538    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/init-textfile/0.log"
Sep 11 12:29:00.601763 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:00.601730    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/node-exporter/0.log"
Sep 11 12:29:00.798628 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:00.798601    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/kube-rbac-proxy/0.log"
Sep 11 12:29:04.199696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:04.199663    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/thanos-query/0.log"
Sep 11 12:29:04.398827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:04.398798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-web/0.log"
Sep 11 12:29:04.599681 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:04.599606    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy/0.log"
Sep 11 12:29:04.805983 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:04.805950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/prom-label-proxy/0.log"
Sep 11 12:29:04.999509 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:04.999480    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-rules/0.log"
Sep 11 12:29:05.200292 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:05.200263    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-metrics/0.log"
Sep 11 12:29:07.002931 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:29:07.002895    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnsxs_ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12/serve-healthcheck-canary/0.log"
Sep 11 12:31:57.548848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:31:57.548822    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 12:32:20.848080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.848047    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cz657"]
Sep 11 12:32:20.850081 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.850064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:20.853133 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.853113    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5n4gp\""
Sep 11 12:32:20.853754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.853739    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 12:32:20.854402 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.854389    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 12:32:20.864252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.864228    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cz657"]
Sep 11 12:32:20.935621 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.935594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77dg\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-kube-api-access-l77dg\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:20.935770 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:20.935628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.036146 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.036118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77dg\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-kube-api-access-l77dg\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.036249 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.036155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.050166 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.050139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77dg\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-kube-api-access-l77dg\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.051666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.051642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51670f95-ca52-4ab9-ab3b-22550ca292a9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cz657\" (UID: \"51670f95-ca52-4ab9-ab3b-22550ca292a9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.158890 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.158832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657"
Sep 11 12:32:21.445951 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.445924    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cz657"]
Sep 11 12:32:21.451594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.451563    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:32:21.898495 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:21.898417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657" event={"ID":"51670f95-ca52-4ab9-ab3b-22550ca292a9","Type":"ContainerStarted","Data":"156c73bb998130042cfa9e93df72da0832c715aa9f0924d50757f12fb6f10379"}
Sep 11 12:32:24.911898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:24.911862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657" event={"ID":"51670f95-ca52-4ab9-ab3b-22550ca292a9","Type":"ContainerStarted","Data":"97a1c9f535d034ebbc6be8926a6345210093e630f62e7b574b3c3431fcc1ba7c"}
Sep 11 12:32:24.933843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:24.933797    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cz657" podStartSLOduration=2.131650694 podStartE2EDuration="4.933780719s" totalImagesPullingTime="2.802130025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:24.932392885 +0000 UTC m=+327.774144030" watchObservedRunningTime="2026-09-11 12:32:24.933780719 +0000 UTC m=+327.775531845"
Sep 11 12:32:32.811360 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.811307    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"]
Sep 11 12:32:32.813538 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.813521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:32.819582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.819562    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 12:32:32.820768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.820747    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:32:32.820865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.820764    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qhr4g\""
Sep 11 12:32:32.827694 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.827673    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"]
Sep 11 12:32:32.920498 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.920473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84210d2-f492-4ccf-9706-222a31f10f2d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:32.920615 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:32.920503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfgk\" (UniqueName: \"kubernetes.io/projected/f84210d2-f492-4ccf-9706-222a31f10f2d-kube-api-access-nnfgk\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.021452 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.021428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84210d2-f492-4ccf-9706-222a31f10f2d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.021588 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.021461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfgk\" (UniqueName: \"kubernetes.io/projected/f84210d2-f492-4ccf-9706-222a31f10f2d-kube-api-access-nnfgk\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.021826 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.021803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84210d2-f492-4ccf-9706-222a31f10f2d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.036416 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.036392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfgk\" (UniqueName: \"kubernetes.io/projected/f84210d2-f492-4ccf-9706-222a31f10f2d-kube-api-access-nnfgk\") pod \"openshift-lws-operator-fd8ccff4c-7bvf9\" (UID: \"f84210d2-f492-4ccf-9706-222a31f10f2d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.121910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.121852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"
Sep 11 12:32:33.402097 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.402068    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9"]
Sep 11 12:32:33.403213 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:32:33.403189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf84210d2_f492_4ccf_9706_222a31f10f2d.slice/crio-a51ba197a10c069e1acf0a94b6500a3e840102085351b9c96f5836ab46269a6a WatchSource:0}: Error finding container a51ba197a10c069e1acf0a94b6500a3e840102085351b9c96f5836ab46269a6a: Status 404 returned error can't find the container with id a51ba197a10c069e1acf0a94b6500a3e840102085351b9c96f5836ab46269a6a
Sep 11 12:32:33.943543 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:33.943509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9" event={"ID":"f84210d2-f492-4ccf-9706-222a31f10f2d","Type":"ContainerStarted","Data":"a51ba197a10c069e1acf0a94b6500a3e840102085351b9c96f5836ab46269a6a"}
Sep 11 12:32:36.957992 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:36.957950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9" event={"ID":"f84210d2-f492-4ccf-9706-222a31f10f2d","Type":"ContainerStarted","Data":"d7e31ee8a986e452da84943b682f40fae1bf0798dbcdedcbe193474b96e4928a"}
Sep 11 12:32:36.979335 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:36.979278    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7bvf9" podStartSLOduration=2.3892779920000002 podStartE2EDuration="4.979264937s" totalImagesPullingTime="2.589986945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:36.977426404 +0000 UTC m=+339.819177533" watchObservedRunningTime="2026-09-11 12:32:36.979264937 +0000 UTC m=+339.821016063"
Sep 11 12:32:55.742622 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.742546    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"]
Sep 11 12:32:55.747494 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.747471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:55.751285 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.751262    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 12:32:55.751902 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.751885    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 12:32:55.752484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.752467    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 12:32:55.752654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.752639    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 12:32:55.752875 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.752861    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d9rvx\""
Sep 11 12:32:55.778421 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.778379    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"]
Sep 11 12:32:55.904275 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.904239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:55.904476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.904284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-tmp\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:55.904476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.904392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-manifests\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:55.904476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.904415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:55.904476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:55.904459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl5p\" (UniqueName: \"kubernetes.io/projected/1c45116d-1144-4aac-b3aa-a2222db72246-kube-api-access-lsl5p\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.005768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.005677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-manifests\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.005768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.005717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.005768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.005743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl5p\" (UniqueName: \"kubernetes.io/projected/1c45116d-1144-4aac-b3aa-a2222db72246-kube-api-access-lsl5p\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.006049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.005778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.006049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.005803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-tmp\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.006154 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.006087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-manifests\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.008193 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.008167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c45116d-1144-4aac-b3aa-a2222db72246-tmp\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.008312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.008284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.008387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.008287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c45116d-1144-4aac-b3aa-a2222db72246-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.022005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.021975    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl5p\" (UniqueName: \"kubernetes.io/projected/1c45116d-1144-4aac-b3aa-a2222db72246-kube-api-access-lsl5p\") pod \"opendatahub-operator-controller-manager-65cbc8d955-g4s6r\" (UID: \"1c45116d-1144-4aac-b3aa-a2222db72246\") " pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.056952 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.056915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:32:56.342188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:56.342110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"]
Sep 11 12:32:56.345918 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:32:56.345881    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c45116d_1144_4aac_b3aa_a2222db72246.slice/crio-cee6b38a949892df9a24c5f8c310d1e74394eecd285df6a2cc45cd4527359fb5 WatchSource:0}: Error finding container cee6b38a949892df9a24c5f8c310d1e74394eecd285df6a2cc45cd4527359fb5: Status 404 returned error can't find the container with id cee6b38a949892df9a24c5f8c310d1e74394eecd285df6a2cc45cd4527359fb5
Sep 11 12:32:57.033003 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:32:57.032955    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r" event={"ID":"1c45116d-1144-4aac-b3aa-a2222db72246","Type":"ContainerStarted","Data":"cee6b38a949892df9a24c5f8c310d1e74394eecd285df6a2cc45cd4527359fb5"}
Sep 11 12:33:00.045895 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:00.045858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r" event={"ID":"1c45116d-1144-4aac-b3aa-a2222db72246","Type":"ContainerStarted","Data":"e41de197600c98883409ca29086b34e8c0c2454578dcf8bbd6e6225c33743fb9"}
Sep 11 12:33:01.051611 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:01.051570    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c45116d-1144-4aac-b3aa-a2222db72246" containerID="e41de197600c98883409ca29086b34e8c0c2454578dcf8bbd6e6225c33743fb9" exitCode=0
Sep 11 12:33:01.052066 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:01.051619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r" event={"ID":"1c45116d-1144-4aac-b3aa-a2222db72246","Type":"ContainerDied","Data":"e41de197600c98883409ca29086b34e8c0c2454578dcf8bbd6e6225c33743fb9"}
Sep 11 12:33:02.058127 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:02.058089    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r" event={"ID":"1c45116d-1144-4aac-b3aa-a2222db72246","Type":"ContainerStarted","Data":"6a1d43f703a35e6e4e77be51dc81c191693b434deae554f0831215e3ff6fe61c"}
Sep 11 12:33:02.058552 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:02.058222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:33:02.091732 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:02.091665    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r" podStartSLOduration=2.391258257 podStartE2EDuration="7.091647122s" totalImagesPullingTime="3.675535887s" totalInitContainerRuntime="1.024852978s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:02.090849144 +0000 UTC m=+364.932600308" watchObservedRunningTime="2026-09-11 12:33:02.091647122 +0000 UTC m=+364.933398249"
Sep 11 12:33:13.065204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:13.065171    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65cbc8d955-g4s6r"
Sep 11 12:33:16.901032 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.900994    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"]
Sep 11 12:33:16.909184 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.909157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:16.913669 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.913649    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 12:33:16.915109 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.915090    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h7zg9\""
Sep 11 12:33:16.915231 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.915134    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 12:33:16.915414 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.915144    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 12:33:16.915414 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.915165    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 12:33:16.926274 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:16.926252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"]
Sep 11 12:33:17.076033 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.075999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbf8\" (UniqueName: \"kubernetes.io/projected/4a7eb3a4-028e-424d-a26f-58a29f2e5826-kube-api-access-hzbf8\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.076198 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.076059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tmp\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.076198 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.076082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tls-certs\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.177334 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.177242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tls-certs\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.177334 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.177292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbf8\" (UniqueName: \"kubernetes.io/projected/4a7eb3a4-028e-424d-a26f-58a29f2e5826-kube-api-access-hzbf8\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.177554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.177354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tmp\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.179533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.179510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tmp\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.179768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.179745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7eb3a4-028e-424d-a26f-58a29f2e5826-tls-certs\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.197744 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.197723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbf8\" (UniqueName: \"kubernetes.io/projected/4a7eb3a4-028e-424d-a26f-58a29f2e5826-kube-api-access-hzbf8\") pod \"kube-auth-proxy-67c8b95cc4-hzmfv\" (UID: \"4a7eb3a4-028e-424d-a26f-58a29f2e5826\") " pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.220741 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.220716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"
Sep 11 12:33:17.519995 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:17.519894    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv"]
Sep 11 12:33:17.522531 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:33:17.522506    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7eb3a4_028e_424d_a26f_58a29f2e5826.slice/crio-8b9fcd060fd1f47850e2234526ce4667ddf782ca6d8fc02f4ad22cfe1aefb910 WatchSource:0}: Error finding container 8b9fcd060fd1f47850e2234526ce4667ddf782ca6d8fc02f4ad22cfe1aefb910: Status 404 returned error can't find the container with id 8b9fcd060fd1f47850e2234526ce4667ddf782ca6d8fc02f4ad22cfe1aefb910
Sep 11 12:33:18.125559 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:18.125520    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv" event={"ID":"4a7eb3a4-028e-424d-a26f-58a29f2e5826","Type":"ContainerStarted","Data":"8b9fcd060fd1f47850e2234526ce4667ddf782ca6d8fc02f4ad22cfe1aefb910"}
Sep 11 12:33:21.144189 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:21.144148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv" event={"ID":"4a7eb3a4-028e-424d-a26f-58a29f2e5826","Type":"ContainerStarted","Data":"7dd155068ad9152000e3192d28f6ea921a61056c976e94ed6daa50627cf6c22a"}
Sep 11 12:33:21.164548 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:21.164486    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67c8b95cc4-hzmfv" podStartSLOduration=2.401792269 podStartE2EDuration="5.164465114s" totalImagesPullingTime="2.762672845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:21.162930329 +0000 UTC m=+384.004681475" watchObservedRunningTime="2026-09-11 12:33:21.164465114 +0000 UTC m=+384.006216241"
Sep 11 12:33:24.494908 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.494871    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"]
Sep 11 12:33:24.498392 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.498367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.501343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.501310    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g4fqd\""
Sep 11 12:33:24.501818 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.501796    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 12:33:24.514932 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.514912    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"]
Sep 11 12:33:24.531001 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.530976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8db6l\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-kube-api-access-8db6l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531347 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531347 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5355b5d-088b-4b2b-ad39-02b8492e1088-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531347 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.531347 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.531266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.631873 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.631844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.631893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.631918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.631963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8db6l\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-kube-api-access-8db6l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.631992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5355b5d-088b-4b2b-ad39-02b8492e1088-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5355b5d-088b-4b2b-ad39-02b8492e1088-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.632963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.632957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.634872 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.634851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.634953 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.634881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.650414 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.650390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.650849 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.650832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8db6l\" (UniqueName: \"kubernetes.io/projected/a5355b5d-088b-4b2b-ad39-02b8492e1088-kube-api-access-8db6l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h\" (UID: \"a5355b5d-088b-4b2b-ad39-02b8492e1088\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:24.808982 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:24.808895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:25.083859 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:25.083792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"]
Sep 11 12:33:25.087003 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:33:25.086977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5355b5d_088b_4b2b_ad39_02b8492e1088.slice/crio-7f7dbdf40b87c34b8864589fdc83cee163ea8a4305ee63ef5ba55badc799c52f WatchSource:0}: Error finding container 7f7dbdf40b87c34b8864589fdc83cee163ea8a4305ee63ef5ba55badc799c52f: Status 404 returned error can't find the container with id 7f7dbdf40b87c34b8864589fdc83cee163ea8a4305ee63ef5ba55badc799c52f
Sep 11 12:33:25.160674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:25.160642    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h" event={"ID":"a5355b5d-088b-4b2b-ad39-02b8492e1088","Type":"ContainerStarted","Data":"7f7dbdf40b87c34b8864589fdc83cee163ea8a4305ee63ef5ba55badc799c52f"}
Sep 11 12:33:27.346776 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.346732    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pjgsm"]
Sep 11 12:33:27.354080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.354049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:27.362448 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.362011    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jx8kg\""
Sep 11 12:33:27.362965 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.361912    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 12:33:27.365769 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.365708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pjgsm"]
Sep 11 12:33:27.452419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.452369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5wd\" (UniqueName: \"kubernetes.io/projected/b8244cc6-7365-4a81-b719-eea621e569c1-kube-api-access-vb5wd\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:27.452581 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.452498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:27.553450 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.553411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5wd\" (UniqueName: \"kubernetes.io/projected/b8244cc6-7365-4a81-b719-eea621e569c1-kube-api-access-vb5wd\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:27.553631 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.553481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:27.553700 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:33:27.553639    2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 12:33:27.553751 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:33:27.553726    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert podName:b8244cc6-7365-4a81-b719-eea621e569c1 nodeName:}" failed. No retries permitted until 2026-09-11 12:33:28.053691102 +0000 UTC m=+390.895442248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert") pod "odh-model-controller-7998c67dc4-pjgsm" (UID: "b8244cc6-7365-4a81-b719-eea621e569c1") : secret "odh-model-controller-webhook-cert" not found
Sep 11 12:33:27.565797 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:27.565737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5wd\" (UniqueName: \"kubernetes.io/projected/b8244cc6-7365-4a81-b719-eea621e569c1-kube-api-access-vb5wd\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:28.058011 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.057957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:28.060734 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.060702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8244cc6-7365-4a81-b719-eea621e569c1-cert\") pod \"odh-model-controller-7998c67dc4-pjgsm\" (UID: \"b8244cc6-7365-4a81-b719-eea621e569c1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:28.269560 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.269531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:28.296582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.296544    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 12:33:28.296705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.296633    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 12:33:28.296705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.296666    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 12:33:28.568510 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:28.568478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pjgsm"]
Sep 11 12:33:28.571484 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:33:28.571443    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8244cc6_7365_4a81_b719_eea621e569c1.slice/crio-48dd41aa77b8f19e5d5a478197dceb0f8a6e1c0b9b8e758e12ee3e32198428fc WatchSource:0}: Error finding container 48dd41aa77b8f19e5d5a478197dceb0f8a6e1c0b9b8e758e12ee3e32198428fc: Status 404 returned error can't find the container with id 48dd41aa77b8f19e5d5a478197dceb0f8a6e1c0b9b8e758e12ee3e32198428fc
Sep 11 12:33:29.182247 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:29.182198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h" event={"ID":"a5355b5d-088b-4b2b-ad39-02b8492e1088","Type":"ContainerStarted","Data":"41dbda79b980e295e3d0dec2fd0dffb10300456110b9a7b67fa26b152e67a6b6"}
Sep 11 12:33:29.183474 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:29.183446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm" event={"ID":"b8244cc6-7365-4a81-b719-eea621e569c1","Type":"ContainerStarted","Data":"48dd41aa77b8f19e5d5a478197dceb0f8a6e1c0b9b8e758e12ee3e32198428fc"}
Sep 11 12:33:29.208813 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:29.208760    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h" podStartSLOduration=2.001381995 podStartE2EDuration="5.208745095s" totalImagesPullingTime="3.2073631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:29.207601814 +0000 UTC m=+392.049352985" watchObservedRunningTime="2026-09-11 12:33:29.208745095 +0000 UTC m=+392.050496220"
Sep 11 12:33:29.809376 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:29.809334    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:29.815971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:29.815941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:30.196363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:30.195685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:30.200356 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:30.197292    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h"
Sep 11 12:33:32.207491 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:32.207428    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm" event={"ID":"b8244cc6-7365-4a81-b719-eea621e569c1","Type":"ContainerStarted","Data":"345275f29a50bb61d89b150b4c5285cd7a7d7febc5eadfaa69775fd41b1ad65d"}
Sep 11 12:33:32.207934 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:32.207726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:32.230958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:32.230906    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm" podStartSLOduration=2.42129946 podStartE2EDuration="5.230890432s" totalImagesPullingTime="2.809590972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:32.22810291 +0000 UTC m=+395.069854059" watchObservedRunningTime="2026-09-11 12:33:32.230890432 +0000 UTC m=+395.072641557"
Sep 11 12:33:34.822596 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.822562    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"]
Sep 11 12:33:34.829206 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.829180    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:34.832716 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.832696    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-27lr8\""
Sep 11 12:33:34.833946 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.833930    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 12:33:34.834057 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.833987    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 12:33:34.834119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.834057    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 12:33:34.840624 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.840602    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"]
Sep 11 12:33:34.920361 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.920313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-metrics-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:34.920491 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.920384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:34.920491 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.920449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mcd\" (UniqueName: \"kubernetes.io/projected/44b835db-4097-4cf0-8840-b3f4744a59bf-kube-api-access-s4mcd\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:34.920491 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:34.920480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b835db-4097-4cf0-8840-b3f4744a59bf-manager-config\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.021041 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.020995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-metrics-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.021201 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.021065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.021201 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.021118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mcd\" (UniqueName: \"kubernetes.io/projected/44b835db-4097-4cf0-8840-b3f4744a59bf-kube-api-access-s4mcd\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.021201 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.021147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b835db-4097-4cf0-8840-b3f4744a59bf-manager-config\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.021943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.021906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44b835db-4097-4cf0-8840-b3f4744a59bf-manager-config\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.023484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.023453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-metrics-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.023850 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.023821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44b835db-4097-4cf0-8840-b3f4744a59bf-cert\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.034711 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.034679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mcd\" (UniqueName: \"kubernetes.io/projected/44b835db-4097-4cf0-8840-b3f4744a59bf-kube-api-access-s4mcd\") pod \"lws-controller-manager-556b8bb6b-r6tn2\" (UID: \"44b835db-4097-4cf0-8840-b3f4744a59bf\") " pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.138488 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.138416    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:35.414134 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:35.414108    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"]
Sep 11 12:33:35.417625 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:33:35.417580    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b835db_4097_4cf0_8840_b3f4744a59bf.slice/crio-fbef05cb93d532991ca7d52f97862f60436e5b7d00fb95db22eab615201910e5 WatchSource:0}: Error finding container fbef05cb93d532991ca7d52f97862f60436e5b7d00fb95db22eab615201910e5: Status 404 returned error can't find the container with id fbef05cb93d532991ca7d52f97862f60436e5b7d00fb95db22eab615201910e5
Sep 11 12:33:36.227811 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:36.227774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2" event={"ID":"44b835db-4097-4cf0-8840-b3f4744a59bf","Type":"ContainerStarted","Data":"fbef05cb93d532991ca7d52f97862f60436e5b7d00fb95db22eab615201910e5"}
Sep 11 12:33:37.233990 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:37.233947    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2" event={"ID":"44b835db-4097-4cf0-8840-b3f4744a59bf","Type":"ContainerStarted","Data":"b484776609467253857a707652d01eac29680062bad8834599deed953d8d26c9"}
Sep 11 12:33:37.234364 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:37.234109    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:33:37.263321 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:37.263271    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2" podStartSLOduration=2.00826939 podStartE2EDuration="3.263254019s" totalImagesPullingTime="1.254984629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:37.263246626 +0000 UTC m=+400.104997775" watchObservedRunningTime="2026-09-11 12:33:37.263254019 +0000 UTC m=+400.105005146"
Sep 11 12:33:43.213355 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:43.213314    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pjgsm"
Sep 11 12:33:48.240308 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:33:48.240278    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-556b8bb6b-r6tn2"
Sep 11 12:35:54.356092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.356015    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:54.359210 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.359191    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:54.363126 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.363105    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mltk2\""
Sep 11 12:35:54.363233 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.363134    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 12:35:54.363582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.363568    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 12:35:54.377904 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.377882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:54.391156 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.391129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vks\" (UniqueName: \"kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks\") pod \"authorino-7bcfd8f9cd-xp524\" (UID: \"326ee4b6-716d-4e5e-abfb-412128828419\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:54.492210 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.492170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vks\" (UniqueName: \"kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks\") pod \"authorino-7bcfd8f9cd-xp524\" (UID: \"326ee4b6-716d-4e5e-abfb-412128828419\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:54.504642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.504615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vks\" (UniqueName: \"kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks\") pod \"authorino-7bcfd8f9cd-xp524\" (UID: \"326ee4b6-716d-4e5e-abfb-412128828419\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:54.668357 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.668274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:54.945119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:54.945087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:54.948941 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:35:54.948920    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod326ee4b6_716d_4e5e_abfb_412128828419.slice/crio-939628dd38f9b4eafbf4bc689cc5d0e3baf228260ae47e558febc85e49d1e743 WatchSource:0}: Error finding container 939628dd38f9b4eafbf4bc689cc5d0e3baf228260ae47e558febc85e49d1e743: Status 404 returned error can't find the container with id 939628dd38f9b4eafbf4bc689cc5d0e3baf228260ae47e558febc85e49d1e743
Sep 11 12:35:55.803312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:55.803274    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" event={"ID":"326ee4b6-716d-4e5e-abfb-412128828419","Type":"ContainerStarted","Data":"939628dd38f9b4eafbf4bc689cc5d0e3baf228260ae47e558febc85e49d1e743"}
Sep 11 12:35:57.549921 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:57.549896    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:57.813729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:57.813639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" event={"ID":"326ee4b6-716d-4e5e-abfb-412128828419","Type":"ContainerStarted","Data":"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"}
Sep 11 12:35:57.813862 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:57.813754    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" podUID="326ee4b6-716d-4e5e-abfb-412128828419" containerName="authorino" containerID="cri-o://e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675" gracePeriod=30
Sep 11 12:35:57.844729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:57.844673    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" podStartSLOduration=1.3055381640000001 podStartE2EDuration="3.84465649s" totalImagesPullingTime="2.539118326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:35:57.843588438 +0000 UTC m=+540.685339578" watchObservedRunningTime="2026-09-11 12:35:57.84465649 +0000 UTC m=+540.686407616"
Sep 11 12:35:58.054367 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.054343    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:58.119616 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.119530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9vks\" (UniqueName: \"kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks\") pod \"326ee4b6-716d-4e5e-abfb-412128828419\" (UID: \"326ee4b6-716d-4e5e-abfb-412128828419\") "
Sep 11 12:35:58.121670 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.121641    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks" (OuterVolumeSpecName: "kube-api-access-l9vks") pod "326ee4b6-716d-4e5e-abfb-412128828419" (UID: "326ee4b6-716d-4e5e-abfb-412128828419"). InnerVolumeSpecName "kube-api-access-l9vks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:35:58.220826 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.220789    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vks\" (UniqueName: \"kubernetes.io/projected/326ee4b6-716d-4e5e-abfb-412128828419-kube-api-access-l9vks\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:35:58.819590 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.819549    2575 generic.go:356] "Generic (PLEG): container finished" podID="326ee4b6-716d-4e5e-abfb-412128828419" containerID="e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675" exitCode=0
Sep 11 12:35:58.819988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.819594    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524"
Sep 11 12:35:58.819988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.819607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" event={"ID":"326ee4b6-716d-4e5e-abfb-412128828419","Type":"ContainerDied","Data":"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"}
Sep 11 12:35:58.819988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.819652    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xp524" event={"ID":"326ee4b6-716d-4e5e-abfb-412128828419","Type":"ContainerDied","Data":"939628dd38f9b4eafbf4bc689cc5d0e3baf228260ae47e558febc85e49d1e743"}
Sep 11 12:35:58.819988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.819673    2575 scope.go:117] "RemoveContainer" containerID="e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"
Sep 11 12:35:58.828218 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.828200    2575 scope.go:117] "RemoveContainer" containerID="e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"
Sep 11 12:35:58.828505 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:35:58.828484    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675\": container with ID starting with e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675 not found: ID does not exist" containerID="e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"
Sep 11 12:35:58.828554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.828514    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675"} err="failed to get container status \"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675\": rpc error: code = NotFound desc = could not find container \"e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675\": container with ID starting with e3684a873636ed61f14af4eb053d3c47165a239477400f80b4b0ba7acd661675 not found: ID does not exist"
Sep 11 12:35:58.847552 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.847519    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:58.848714 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:58.848687    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xp524"]
Sep 11 12:35:59.630033 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:35:59.629994    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326ee4b6-716d-4e5e-abfb-412128828419" path="/var/lib/kubelet/pods/326ee4b6-716d-4e5e-abfb-412128828419/volumes"
Sep 11 12:37:07.063071 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.063037    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"]
Sep 11 12:37:07.063572 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.063385    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326ee4b6-716d-4e5e-abfb-412128828419" containerName="authorino"
Sep 11 12:37:07.063572 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.063398    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="326ee4b6-716d-4e5e-abfb-412128828419" containerName="authorino"
Sep 11 12:37:07.063572 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.063463    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="326ee4b6-716d-4e5e-abfb-412128828419" containerName="authorino"
Sep 11 12:37:07.066195 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.066176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.069664 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.069644    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wj8cl\""
Sep 11 12:37:07.074544 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.074524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 12:37:07.089993 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.089965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"]
Sep 11 12:37:07.154389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.154352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.154521 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.154445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7q9m\" (UniqueName: \"kubernetes.io/projected/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-kube-api-access-c7q9m\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.255151 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.255121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7q9m\" (UniqueName: \"kubernetes.io/projected/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-kube-api-access-c7q9m\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.255390 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.255181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.255837 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.255816    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.263741 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.263719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7q9m\" (UniqueName: \"kubernetes.io/projected/d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6-kube-api-access-c7q9m\") pod \"payload-processing-5b54b7fbb7-zzfmq\" (UID: \"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.376279 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.376220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:07.647615 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:07.647592    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"]
Sep 11 12:37:07.649742 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:37:07.649713    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e3eecf_bfdc_4886_a890_ba8f58f5a8b6.slice/crio-05142ba8a567b34a213e0b03d9c9c07824e037d66bf4dfdb7219392a69baa45d WatchSource:0}: Error finding container 05142ba8a567b34a213e0b03d9c9c07824e037d66bf4dfdb7219392a69baa45d: Status 404 returned error can't find the container with id 05142ba8a567b34a213e0b03d9c9c07824e037d66bf4dfdb7219392a69baa45d
Sep 11 12:37:08.112376 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:08.112340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq" event={"ID":"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6","Type":"ContainerStarted","Data":"05142ba8a567b34a213e0b03d9c9c07824e037d66bf4dfdb7219392a69baa45d"}
Sep 11 12:37:10.122177 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:10.122138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq" event={"ID":"d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6","Type":"ContainerStarted","Data":"bc2e4becdf252a50368240c0ebfdc0cde44470883680c858a83f6c2b0fa6ac8e"}
Sep 11 12:37:10.122654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:10.122272    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:10.140537 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:10.140490    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq" podStartSLOduration=1.449499562 podStartE2EDuration="3.140475163s" totalImagesPullingTime="1.690975601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:37:10.13857586 +0000 UTC m=+612.980327013" watchObservedRunningTime="2026-09-11 12:37:10.140475163 +0000 UTC m=+612.982226290"
Sep 11 12:37:21.128517 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:21.128486    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzfmq"
Sep 11 12:37:49.574815 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.574775    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"]
Sep 11 12:37:49.577685 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.577662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.581707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.581676    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 12:37:49.581707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.581677    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-67t6s\""
Sep 11 12:37:49.581895 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.581676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 12:37:49.581895 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.581685    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 12:37:49.587910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.587883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"]
Sep 11 12:37:49.696537 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.696700 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.696700 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.696700 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef48ca8-dc30-4471-a565-7100af08435f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.696874 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.696929 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.696891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9gm\" (UniqueName: \"kubernetes.io/projected/2ef48ca8-dc30-4471-a565-7100af08435f-kube-api-access-bs9gm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797619 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797619 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef48ca8-dc30-4471-a565-7100af08435f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797619 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797619 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9gm\" (UniqueName: \"kubernetes.io/projected/2ef48ca8-dc30-4471-a565-7100af08435f-kube-api-access-bs9gm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.797956 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.798067 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.798067 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.797979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.799800 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.799779    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef48ca8-dc30-4471-a565-7100af08435f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.800050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.800033    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef48ca8-dc30-4471-a565-7100af08435f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.806858 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.806839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9gm\" (UniqueName: \"kubernetes.io/projected/2ef48ca8-dc30-4471-a565-7100af08435f-kube-api-access-bs9gm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm\" (UID: \"2ef48ca8-dc30-4471-a565-7100af08435f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:49.890898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:49.890826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:50.160808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:50.160787    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"]
Sep 11 12:37:50.163167 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:37:50.163140    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef48ca8_dc30_4471_a565_7100af08435f.slice/crio-e351bfe8f64c2f615e480b615bc63c36c83f5e07841edc3c6e52fa1cb2994ff6 WatchSource:0}: Error finding container e351bfe8f64c2f615e480b615bc63c36c83f5e07841edc3c6e52fa1cb2994ff6: Status 404 returned error can't find the container with id e351bfe8f64c2f615e480b615bc63c36c83f5e07841edc3c6e52fa1cb2994ff6
Sep 11 12:37:50.164552 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:50.164533    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:37:50.291124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:50.291088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm" event={"ID":"2ef48ca8-dc30-4471-a565-7100af08435f","Type":"ContainerStarted","Data":"e351bfe8f64c2f615e480b615bc63c36c83f5e07841edc3c6e52fa1cb2994ff6"}
Sep 11 12:37:51.296746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:51.296712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm" event={"ID":"2ef48ca8-dc30-4471-a565-7100af08435f","Type":"ContainerStarted","Data":"1967ab5e60fa46c8e69c9ba5a8942666d28d8c3c0cc05f84c10c617bc872d189"}
Sep 11 12:37:51.317937 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:51.317889    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm" podStartSLOduration=1.404802225 podStartE2EDuration="2.317873968s" totalImagesPullingTime="913.071743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:37:51.316406666 +0000 UTC m=+654.158157807" watchObservedRunningTime="2026-09-11 12:37:51.317873968 +0000 UTC m=+654.159625093"
Sep 11 12:37:59.891802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:59.891763    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:59.892272 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:59.891838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:37:59.903986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:37:59.903960    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:38:00.344561 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:00.344529    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm"
Sep 11 12:38:02.395377 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.395346    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"]
Sep 11 12:38:02.397727 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.397702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.401050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.401030    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 12:38:02.411002 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.410978    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"]
Sep 11 12:38:02.505017 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.504977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.505185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.505029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.505185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.505084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.505185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.505140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.505185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.505168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhz2z\" (UniqueName: \"kubernetes.io/projected/8591664a-e618-4d86-994c-3292349cbc98-kube-api-access-fhz2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.505319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.505215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591664a-e618-4d86-994c-3292349cbc98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.605958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.605921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.605958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.605962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhz2z\" (UniqueName: \"kubernetes.io/projected/8591664a-e618-4d86-994c-3292349cbc98-kube-api-access-fhz2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606173 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591664a-e618-4d86-994c-3292349cbc98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606249 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606249 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606353    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.606638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.606618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.608395 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.608378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591664a-e618-4d86-994c-3292349cbc98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.608690 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.608671    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591664a-e618-4d86-994c-3292349cbc98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.614125 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.614099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhz2z\" (UniqueName: \"kubernetes.io/projected/8591664a-e618-4d86-994c-3292349cbc98-kube-api-access-fhz2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xqhf4\" (UID: \"8591664a-e618-4d86-994c-3292349cbc98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.707608 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.707573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:02.981646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:02.981619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"]
Sep 11 12:38:02.983795 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:38:02.983766    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8591664a_e618_4d86_994c_3292349cbc98.slice/crio-64d26faff1946c973c3abdd96b98504d1a4df9d8d80ba46f1921de17f29272e9 WatchSource:0}: Error finding container 64d26faff1946c973c3abdd96b98504d1a4df9d8d80ba46f1921de17f29272e9: Status 404 returned error can't find the container with id 64d26faff1946c973c3abdd96b98504d1a4df9d8d80ba46f1921de17f29272e9
Sep 11 12:38:03.346615 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:03.346531    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4" event={"ID":"8591664a-e618-4d86-994c-3292349cbc98","Type":"ContainerStarted","Data":"47d246c882f66561f1f8445823f25bba5042d32b2d8b7f6a45bd59a9966aa88a"}
Sep 11 12:38:03.346615 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:03.346589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4" event={"ID":"8591664a-e618-4d86-994c-3292349cbc98","Type":"ContainerStarted","Data":"64d26faff1946c973c3abdd96b98504d1a4df9d8d80ba46f1921de17f29272e9"}
Sep 11 12:38:03.364602 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:03.364543    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4" podStartSLOduration=1.1698209880000001 podStartE2EDuration="1.364523458s" totalImagesPullingTime="194.70247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:03.363083177 +0000 UTC m=+666.204834326" watchObservedRunningTime="2026-09-11 12:38:03.364523458 +0000 UTC m=+666.206274587"
Sep 11 12:38:10.080733 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.080699    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"]
Sep 11 12:38:10.083334 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.083306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.086136 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.086112    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 12:38:10.093469 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.093443    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"]
Sep 11 12:38:10.168764 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwcg\" (UniqueName: \"kubernetes.io/projected/131fa2c8-be72-48d7-9563-34275474b9d3-kube-api-access-6nwcg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.168920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.168920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.168920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.168920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131fa2c8-be72-48d7-9563-34275474b9d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.168920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.168871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269276 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwcg\" (UniqueName: \"kubernetes.io/projected/131fa2c8-be72-48d7-9563-34275474b9d3-kube-api-access-6nwcg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269495 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269495 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269495 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269495 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131fa2c8-be72-48d7-9563-34275474b9d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269712 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269769 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269825 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.269908 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.269880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.271575 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.271547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131fa2c8-be72-48d7-9563-34275474b9d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.271870 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.271852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131fa2c8-be72-48d7-9563-34275474b9d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.277519 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.277498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwcg\" (UniqueName: \"kubernetes.io/projected/131fa2c8-be72-48d7-9563-34275474b9d3-kube-api-access-6nwcg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l\" (UID: \"131fa2c8-be72-48d7-9563-34275474b9d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.393887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.393807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:10.668062 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:10.668033    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"]
Sep 11 12:38:10.670485 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:38:10.670456    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131fa2c8_be72_48d7_9563_34275474b9d3.slice/crio-873e10958482795c7482afdaf1f7bbe4783ed4d4181709a28fb407a3d6a03e13 WatchSource:0}: Error finding container 873e10958482795c7482afdaf1f7bbe4783ed4d4181709a28fb407a3d6a03e13: Status 404 returned error can't find the container with id 873e10958482795c7482afdaf1f7bbe4783ed4d4181709a28fb407a3d6a03e13
Sep 11 12:38:11.390096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:11.390059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l" event={"ID":"131fa2c8-be72-48d7-9563-34275474b9d3","Type":"ContainerStarted","Data":"20110c639efd37b0adfee45974c47603d373e8de16d60f662e004b9ad58c2f1f"}
Sep 11 12:38:11.390498 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:11.390116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l" event={"ID":"131fa2c8-be72-48d7-9563-34275474b9d3","Type":"ContainerStarted","Data":"873e10958482795c7482afdaf1f7bbe4783ed4d4181709a28fb407a3d6a03e13"}
Sep 11 12:38:11.410279 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:11.410222    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l" podStartSLOduration=1.22631966 podStartE2EDuration="1.410206009s" totalImagesPullingTime="183.886349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:11.408792237 +0000 UTC m=+674.250543400" watchObservedRunningTime="2026-09-11 12:38:11.410206009 +0000 UTC m=+674.251957136"
Sep 11 12:38:12.708007 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:12.707956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:12.708423 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:12.708042    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:12.720415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:12.720392    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:13.409472 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:13.409444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xqhf4"
Sep 11 12:38:20.394671 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:20.394618    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:20.395154 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:20.394696    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:20.407122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:20.407094    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:20.445159 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:20.445131    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l"
Sep 11 12:38:39.286701 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.286665    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"]
Sep 11 12:38:39.296650 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.296622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.299280 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.299255    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 12:38:39.304445 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.304066    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"]
Sep 11 12:38:39.411801 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.411764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.411801 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.411806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.412081 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.411829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9639b-3bac-4804-8475-ef092fb1f18f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.412081 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.411901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.412081 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.411952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkvc\" (UniqueName: \"kubernetes.io/projected/c8d9639b-3bac-4804-8475-ef092fb1f18f-kube-api-access-4kkvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.412081 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.412000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513157 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9639b-3bac-4804-8475-ef092fb1f18f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkvc\" (UniqueName: \"kubernetes.io/projected/c8d9639b-3bac-4804-8475-ef092fb1f18f-kube-api-access-4kkvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513665    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513877 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.513988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.513921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.515687 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.515665    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d9639b-3bac-4804-8475-ef092fb1f18f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.515857 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.515838    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9639b-3bac-4804-8475-ef092fb1f18f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.522262 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.522235    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkvc\" (UniqueName: \"kubernetes.io/projected/c8d9639b-3bac-4804-8475-ef092fb1f18f-kube-api-access-4kkvc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx\" (UID: \"c8d9639b-3bac-4804-8475-ef092fb1f18f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.608396 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.608301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:39.882633 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:39.882608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"]
Sep 11 12:38:39.884766 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:38:39.884739    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8d9639b_3bac_4804_8475_ef092fb1f18f.slice/crio-9b937b6bcfcee11aa2ed9e47b820151057a55a838819aa6e5e6560770d2c063c WatchSource:0}: Error finding container 9b937b6bcfcee11aa2ed9e47b820151057a55a838819aa6e5e6560770d2c063c: Status 404 returned error can't find the container with id 9b937b6bcfcee11aa2ed9e47b820151057a55a838819aa6e5e6560770d2c063c
Sep 11 12:38:40.524297 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:40.524258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx" event={"ID":"c8d9639b-3bac-4804-8475-ef092fb1f18f","Type":"ContainerStarted","Data":"6d0db875f356257416099c40a3ac3d40ca63ac8c8e6238f9eafb3852781e6fcd"}
Sep 11 12:38:40.524297 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:40.524300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx" event={"ID":"c8d9639b-3bac-4804-8475-ef092fb1f18f","Type":"ContainerStarted","Data":"9b937b6bcfcee11aa2ed9e47b820151057a55a838819aa6e5e6560770d2c063c"}
Sep 11 12:38:40.544684 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:40.544634    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx" podStartSLOduration=1.362091396 podStartE2EDuration="1.54461766s" totalImagesPullingTime="182.526264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:40.54291821 +0000 UTC m=+703.384669356" watchObservedRunningTime="2026-09-11 12:38:40.54461766 +0000 UTC m=+703.386368786"
Sep 11 12:38:49.608889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:49.608844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:49.609400 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:49.609120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:49.621551 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:49.621524    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:38:50.580907 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:38:50.580827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx"
Sep 11 12:40:08.710813 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.710771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:40:08.714172 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.714153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.717793 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.717760    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edce0b\""
Sep 11 12:40:08.718869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.718848    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-edce0b-dockercfg-n4mx5\""
Sep 11 12:40:08.731154 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.731129    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:40:08.814954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.814919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.815147 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.814969    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzwz\" (UniqueName: \"kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.915942 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.915903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.916106 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.915958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzwz\" (UniqueName: \"kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.916610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.916591    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:08.924765 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:08.924737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzwz\" (UniqueName: \"kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz\") pod \"payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:09.024744 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.024650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:09.297754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.297657    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:40:09.300616 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:09.300582    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5168fd_6559_47d3_b798_3b812e4db5ca.slice/crio-4438ddff5ef5017728595982b0fc257080c89c8be0950a5b6da6ff4cbb017747 WatchSource:0}: Error finding container 4438ddff5ef5017728595982b0fc257080c89c8be0950a5b6da6ff4cbb017747: Status 404 returned error can't find the container with id 4438ddff5ef5017728595982b0fc257080c89c8be0950a5b6da6ff4cbb017747
Sep 11 12:40:09.912821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.912775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" event={"ID":"4c5168fd-6559-47d3-b798-3b812e4db5ca","Type":"ContainerStarted","Data":"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"}
Sep 11 12:40:09.913246 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.912840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" event={"ID":"4c5168fd-6559-47d3-b798-3b812e4db5ca","Type":"ContainerStarted","Data":"4438ddff5ef5017728595982b0fc257080c89c8be0950a5b6da6ff4cbb017747"}
Sep 11 12:40:09.913246 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.912903    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:09.931207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:09.931158    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" podStartSLOduration=1.93114121 podStartE2EDuration="1.93114121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:09.929573335 +0000 UTC m=+792.771324493" watchObservedRunningTime="2026-09-11 12:40:09.93114121 +0000 UTC m=+792.772892335"
Sep 11 12:40:16.842840 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.842798    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:40:16.846569 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.846545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:16.849801 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.849782    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d7de9d8\""
Sep 11 12:40:16.850315 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.850298    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d7de9d8-dockercfg-hhjn7\""
Sep 11 12:40:16.865397 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.865367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:40:16.947806 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.947771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:40:16.950971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.950955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:16.967064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.967038    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:40:16.993368 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.993310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:16.993507 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:16.993445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kv8c\" (UniqueName: \"kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.094352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.094245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tl2t\" (UniqueName: \"kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.094352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.094287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.094533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.094381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kv8c\" (UniqueName: \"kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.094533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.094403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.094997 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.094979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.113685 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.113651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kv8c\" (UniqueName: \"kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c\") pod \"payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") " pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.156393 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.156357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.195550 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.195511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tl2t\" (UniqueName: \"kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.195743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.195612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.196360 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.196341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.216016 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.213472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tl2t\" (UniqueName: \"kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t\") pod \"payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.260732 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.260700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.504368 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:17.504316    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc732c8cf_9e56_4cfb_901d_a5b44864b067.slice/crio-62d72bca2d7548aae9aeeb4b22f53d730b4ea80a66163c28718f9783b96c0716 WatchSource:0}: Error finding container 62d72bca2d7548aae9aeeb4b22f53d730b4ea80a66163c28718f9783b96c0716: Status 404 returned error can't find the container with id 62d72bca2d7548aae9aeeb4b22f53d730b4ea80a66163c28718f9783b96c0716
Sep 11 12:40:17.505127 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.505072    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:40:17.551267 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:17.551226    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc130c670_8d53_4382_8f48_b9b31758e5c9.slice/crio-a604982f2a92c8a7938e6527e34c8c1c48cac67294fe6c8a0eaa5c62c4fe82c5 WatchSource:0}: Error finding container a604982f2a92c8a7938e6527e34c8c1c48cac67294fe6c8a0eaa5c62c4fe82c5: Status 404 returned error can't find the container with id a604982f2a92c8a7938e6527e34c8c1c48cac67294fe6c8a0eaa5c62c4fe82c5
Sep 11 12:40:17.554236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.554204    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:40:17.952654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.952620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" event={"ID":"c130c670-8d53-4382-8f48-b9b31758e5c9","Type":"ContainerStarted","Data":"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"}
Sep 11 12:40:17.953110 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.952688    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:17.953110 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.952715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" event={"ID":"c130c670-8d53-4382-8f48-b9b31758e5c9","Type":"ContainerStarted","Data":"a604982f2a92c8a7938e6527e34c8c1c48cac67294fe6c8a0eaa5c62c4fe82c5"}
Sep 11 12:40:17.954511 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.954484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" event={"ID":"c732c8cf-9e56-4cfb-901d-a5b44864b067","Type":"ContainerStarted","Data":"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"}
Sep 11 12:40:17.954657 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.954533    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" event={"ID":"c732c8cf-9e56-4cfb-901d-a5b44864b067","Type":"ContainerStarted","Data":"62d72bca2d7548aae9aeeb4b22f53d730b4ea80a66163c28718f9783b96c0716"}
Sep 11 12:40:17.954657 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.954556    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:17.996823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:17.996767    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" podStartSLOduration=1.996752437 podStartE2EDuration="1.996752437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:17.99566075 +0000 UTC m=+800.837411895" watchObservedRunningTime="2026-09-11 12:40:17.996752437 +0000 UTC m=+800.838503657"
Sep 11 12:40:18.039756 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.039701    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" podStartSLOduration=2.039683616 podStartE2EDuration="2.039683616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:18.038544299 +0000 UTC m=+800.880295444" watchObservedRunningTime="2026-09-11 12:40:18.039683616 +0000 UTC m=+800.881434741"
Sep 11 12:40:18.602827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.602788    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:40:18.606466 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.606449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.609167 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.609146    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9878a9\""
Sep 11 12:40:18.618007 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.617973    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:40:18.710922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.710882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.710922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.710932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l59k\" (UniqueName: \"kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.739321 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.739288    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:40:18.742586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.742570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.745462 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.745434    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9878a9-dockercfg-666k8\""
Sep 11 12:40:18.757839 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.757814    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:40:18.812020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.811982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l59k\" (UniqueName: \"kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.812213 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.812050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqvp\" (UniqueName: \"kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.812213 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.812117    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.812213 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.812164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.812758 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.812740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.822766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.822744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l59k\" (UniqueName: \"kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k\") pod \"payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.913319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.913217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.913319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.913303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqvp\" (UniqueName: \"kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.913940 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.913921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:18.917121 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.917097    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:18.923715 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:18.923686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqvp\" (UniqueName: \"kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp\") pod \"payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") " pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:19.053287 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.053246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:19.193614 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.193588    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:40:19.196351 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:19.196299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8955d5e_a652_45aa_b836_63e6a29106b1.slice/crio-7e08fcde9adfa84e75af78ca976d52792cb68b7d094fa5fabf29909b4b2512b9 WatchSource:0}: Error finding container 7e08fcde9adfa84e75af78ca976d52792cb68b7d094fa5fabf29909b4b2512b9: Status 404 returned error can't find the container with id 7e08fcde9adfa84e75af78ca976d52792cb68b7d094fa5fabf29909b4b2512b9
Sep 11 12:40:19.333189 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.333163    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:40:19.335246 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:19.335219    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1d6cb5_51e2_4824_8e69_61c0496a2e3e.slice/crio-bda87a7aecd531f0e4438365a36eb185c8442c9417159e4660299c30a644b037 WatchSource:0}: Error finding container bda87a7aecd531f0e4438365a36eb185c8442c9417159e4660299c30a644b037: Status 404 returned error can't find the container with id bda87a7aecd531f0e4438365a36eb185c8442c9417159e4660299c30a644b037
Sep 11 12:40:19.966430 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.966386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" event={"ID":"c8955d5e-a652-45aa-b836-63e6a29106b1","Type":"ContainerStarted","Data":"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"}
Sep 11 12:40:19.966613 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.966453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" event={"ID":"c8955d5e-a652-45aa-b836-63e6a29106b1","Type":"ContainerStarted","Data":"7e08fcde9adfa84e75af78ca976d52792cb68b7d094fa5fabf29909b4b2512b9"}
Sep 11 12:40:19.966613 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.966496    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:19.968114 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.968084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" event={"ID":"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e","Type":"ContainerStarted","Data":"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"}
Sep 11 12:40:19.968250 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.968126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" event={"ID":"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e","Type":"ContainerStarted","Data":"bda87a7aecd531f0e4438365a36eb185c8442c9417159e4660299c30a644b037"}
Sep 11 12:40:19.968250 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.968146    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:19.990455 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:19.990405    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" podStartSLOduration=1.990387143 podStartE2EDuration="1.990387143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:19.9878301 +0000 UTC m=+802.829581238" watchObservedRunningTime="2026-09-11 12:40:19.990387143 +0000 UTC m=+802.832138271"
Sep 11 12:40:20.006825 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:20.006776    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" podStartSLOduration=2.00676045 podStartE2EDuration="2.00676045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:20.006178094 +0000 UTC m=+802.847929242" watchObservedRunningTime="2026-09-11 12:40:20.00676045 +0000 UTC m=+802.848511576"
Sep 11 12:40:20.919785 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:20.919750    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:40:28.962389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:28.962353    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:40:28.962757 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:28.962423    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:40:30.977523 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:30.977487    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:40:30.977909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:30.977709    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:40:38.462711 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.462672    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:40:38.465458 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.465441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.468772 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.468751    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f06aba\""
Sep 11 12:40:38.476620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.476583    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:40:38.480931 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.480902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.481028 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.480964    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.581918 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.581883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.582116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.581936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.582629 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.582606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.591706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.591681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls\") pod \"payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:38.776545 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:38.776467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:39.051449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:39.051396    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:40:39.053857 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:40:39.053827    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad600284_95bd_475f_9094_72a478483f53.slice/crio-288bf453f660b091d21bd341fcc832063fdf4e7c68be69daa4246ed396b12900 WatchSource:0}: Error finding container 288bf453f660b091d21bd341fcc832063fdf4e7c68be69daa4246ed396b12900: Status 404 returned error can't find the container with id 288bf453f660b091d21bd341fcc832063fdf4e7c68be69daa4246ed396b12900
Sep 11 12:40:40.063149 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:40.063111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" event={"ID":"ad600284-95bd-475f-9094-72a478483f53","Type":"ContainerStarted","Data":"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"}
Sep 11 12:40:40.063546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:40.063169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" event={"ID":"ad600284-95bd-475f-9094-72a478483f53","Type":"ContainerStarted","Data":"288bf453f660b091d21bd341fcc832063fdf4e7c68be69daa4246ed396b12900"}
Sep 11 12:40:40.063546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:40.063232    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:40:40.081166 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:40.081114    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" podStartSLOduration=2.08110084 podStartE2EDuration="2.08110084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:40.079479151 +0000 UTC m=+822.921230276" watchObservedRunningTime="2026-09-11 12:40:40.08110084 +0000 UTC m=+822.922851964"
Sep 11 12:40:51.069871 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:40:51.069841    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:41:16.456369 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.456318    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:41:16.456848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.456631    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" podUID="c732c8cf-9e56-4cfb-901d-a5b44864b067" containerName="payload-processing" containerID="cri-o://18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea" gracePeriod=30
Sep 11 12:41:16.462781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.462747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:41:16.463061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.463014    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" podUID="c130c670-8d53-4382-8f48-b9b31758e5c9" containerName="payload-pre-processing" containerID="cri-o://63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e" gracePeriod=30
Sep 11 12:41:16.740892 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.740867    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:41:16.744225 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.744196    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:41:16.934646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.934607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume\") pod \"c732c8cf-9e56-4cfb-901d-a5b44864b067\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") "
Sep 11 12:41:16.934646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.934652    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tl2t\" (UniqueName: \"kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t\") pod \"c130c670-8d53-4382-8f48-b9b31758e5c9\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") "
Sep 11 12:41:16.934890 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.934680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kv8c\" (UniqueName: \"kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c\") pod \"c732c8cf-9e56-4cfb-901d-a5b44864b067\" (UID: \"c732c8cf-9e56-4cfb-901d-a5b44864b067\") "
Sep 11 12:41:16.934890 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.934789    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume\") pod \"c130c670-8d53-4382-8f48-b9b31758e5c9\" (UID: \"c130c670-8d53-4382-8f48-b9b31758e5c9\") "
Sep 11 12:41:16.935136 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.935113    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c732c8cf-9e56-4cfb-901d-a5b44864b067" (UID: "c732c8cf-9e56-4cfb-901d-a5b44864b067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:41:16.935203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.935159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c130c670-8d53-4382-8f48-b9b31758e5c9" (UID: "c130c670-8d53-4382-8f48-b9b31758e5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:41:16.936972 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.936949    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c" (OuterVolumeSpecName: "kube-api-access-4kv8c") pod "c732c8cf-9e56-4cfb-901d-a5b44864b067" (UID: "c732c8cf-9e56-4cfb-901d-a5b44864b067"). InnerVolumeSpecName "kube-api-access-4kv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:16.937049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:16.936978    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t" (OuterVolumeSpecName: "kube-api-access-4tl2t") pod "c130c670-8d53-4382-8f48-b9b31758e5c9" (UID: "c130c670-8d53-4382-8f48-b9b31758e5c9"). InnerVolumeSpecName "kube-api-access-4tl2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:17.035824 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.035740    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c130c670-8d53-4382-8f48-b9b31758e5c9-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:17.035824 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.035770    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c732c8cf-9e56-4cfb-901d-a5b44864b067-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:17.035824 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.035780    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tl2t\" (UniqueName: \"kubernetes.io/projected/c130c670-8d53-4382-8f48-b9b31758e5c9-kube-api-access-4tl2t\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:17.035824 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.035789    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kv8c\" (UniqueName: \"kubernetes.io/projected/c732c8cf-9e56-4cfb-901d-a5b44864b067-kube-api-access-4kv8c\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:17.238177 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.238136    2575 generic.go:356] "Generic (PLEG): container finished" podID="c732c8cf-9e56-4cfb-901d-a5b44864b067" containerID="18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea" exitCode=0
Sep 11 12:41:17.238386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.238184    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"
Sep 11 12:41:17.238386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.238225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" event={"ID":"c732c8cf-9e56-4cfb-901d-a5b44864b067","Type":"ContainerDied","Data":"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"}
Sep 11 12:41:17.238386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.238284    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr" event={"ID":"c732c8cf-9e56-4cfb-901d-a5b44864b067","Type":"ContainerDied","Data":"62d72bca2d7548aae9aeeb4b22f53d730b4ea80a66163c28718f9783b96c0716"}
Sep 11 12:41:17.238386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.238315    2575 scope.go:117] "RemoveContainer" containerID="18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"
Sep 11 12:41:17.239678 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.239659    2575 generic.go:356] "Generic (PLEG): container finished" podID="c130c670-8d53-4382-8f48-b9b31758e5c9" containerID="63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e" exitCode=0
Sep 11 12:41:17.239746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.239695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" event={"ID":"c130c670-8d53-4382-8f48-b9b31758e5c9","Type":"ContainerDied","Data":"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"}
Sep 11 12:41:17.239746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.239709    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"
Sep 11 12:41:17.239746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.239724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb" event={"ID":"c130c670-8d53-4382-8f48-b9b31758e5c9","Type":"ContainerDied","Data":"a604982f2a92c8a7938e6527e34c8c1c48cac67294fe6c8a0eaa5c62c4fe82c5"}
Sep 11 12:41:17.248818 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.248792    2575 scope.go:117] "RemoveContainer" containerID="18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"
Sep 11 12:41:17.249213 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:41:17.249191    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea\": container with ID starting with 18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea not found: ID does not exist" containerID="18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"
Sep 11 12:41:17.249304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.249226    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea"} err="failed to get container status \"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea\": rpc error: code = NotFound desc = could not find container \"18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea\": container with ID starting with 18a289e88b5901f1d418547f01c5e90afd0e979f47a310bb72c28ae583e3f7ea not found: ID does not exist"
Sep 11 12:41:17.249304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.249249    2575 scope.go:117] "RemoveContainer" containerID="63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"
Sep 11 12:41:17.257183 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.257165    2575 scope.go:117] "RemoveContainer" containerID="63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"
Sep 11 12:41:17.257450 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:41:17.257431    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e\": container with ID starting with 63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e not found: ID does not exist" containerID="63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"
Sep 11 12:41:17.257500 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.257457    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e"} err="failed to get container status \"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e\": rpc error: code = NotFound desc = could not find container \"63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e\": container with ID starting with 63f93c9752753c450cff99ca1bc0d42ae963c305d32a47df167d933d9dbee95e not found: ID does not exist"
Sep 11 12:41:17.265742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.265720    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:41:17.268571 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.268548    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d7de9d8-d5444d5c-k6grr"]
Sep 11 12:41:17.287430 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.287364    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:41:17.291509 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.291487    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d7de9d8-7c77b6bf78-pmlfb"]
Sep 11 12:41:17.630122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.630048    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c130c670-8d53-4382-8f48-b9b31758e5c9" path="/var/lib/kubelet/pods/c130c670-8d53-4382-8f48-b9b31758e5c9/volumes"
Sep 11 12:41:17.630470 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:17.630438    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c732c8cf-9e56-4cfb-901d-a5b44864b067" path="/var/lib/kubelet/pods/c732c8cf-9e56-4cfb-901d-a5b44864b067/volumes"
Sep 11 12:41:20.909724 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.909685    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:20.910119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910068    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c130c670-8d53-4382-8f48-b9b31758e5c9" containerName="payload-pre-processing"
Sep 11 12:41:20.910119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910080    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c130c670-8d53-4382-8f48-b9b31758e5c9" containerName="payload-pre-processing"
Sep 11 12:41:20.910119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910095    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c732c8cf-9e56-4cfb-901d-a5b44864b067" containerName="payload-processing"
Sep 11 12:41:20.910119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910100    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c732c8cf-9e56-4cfb-901d-a5b44864b067" containerName="payload-processing"
Sep 11 12:41:20.910309 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910158    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c130c670-8d53-4382-8f48-b9b31758e5c9" containerName="payload-pre-processing"
Sep 11 12:41:20.910309 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.910168    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c732c8cf-9e56-4cfb-901d-a5b44864b067" containerName="payload-processing"
Sep 11 12:41:20.914595 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.914577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:20.917605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.917585    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-47131dfa-dockercfg-zh9sl\""
Sep 11 12:41:20.917707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.917627    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-47131dfa\""
Sep 11 12:41:20.923561 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.923537    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:20.970319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.970281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:20.970481 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:20.970414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkw2s\" (UniqueName: \"kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.071050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.071013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkw2s\" (UniqueName: \"kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.071230 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.071099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.071857 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.071837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.080827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.080802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkw2s\" (UniqueName: \"kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s\") pod \"payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.225971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.225931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:21.504620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:21.504387    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:21.506835 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:41:21.506788    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a865ee_3cce_4990_90cc_521595d38449.slice/crio-56037eda38c0df0612c1efc7bb23aa26b1160d1bdfd74238e2160a5074e4754f WatchSource:0}: Error finding container 56037eda38c0df0612c1efc7bb23aa26b1160d1bdfd74238e2160a5074e4754f: Status 404 returned error can't find the container with id 56037eda38c0df0612c1efc7bb23aa26b1160d1bdfd74238e2160a5074e4754f
Sep 11 12:41:22.268176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:22.268139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" event={"ID":"78a865ee-3cce-4990-90cc-521595d38449","Type":"ContainerStarted","Data":"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"}
Sep 11 12:41:22.268658 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:22.268192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" event={"ID":"78a865ee-3cce-4990-90cc-521595d38449","Type":"ContainerStarted","Data":"56037eda38c0df0612c1efc7bb23aa26b1160d1bdfd74238e2160a5074e4754f"}
Sep 11 12:41:22.268658 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:22.268297    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:22.287407 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:22.287356    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" podStartSLOduration=2.2873402990000002 podStartE2EDuration="2.287340299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:41:22.285779366 +0000 UTC m=+865.127530520" watchObservedRunningTime="2026-09-11 12:41:22.287340299 +0000 UTC m=+865.129091416"
Sep 11 12:41:33.276506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:33.276472    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:51.556129 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.556033    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:51.556616 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.556300    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" podUID="78a865ee-3cce-4990-90cc-521595d38449" containerName="payload-pre-processing" containerID="cri-o://d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6" gracePeriod=30
Sep 11 12:41:51.800899 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.800874    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:51.948163 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.948135    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume\") pod \"78a865ee-3cce-4990-90cc-521595d38449\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") "
Sep 11 12:41:51.948311 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.948240    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkw2s\" (UniqueName: \"kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s\") pod \"78a865ee-3cce-4990-90cc-521595d38449\" (UID: \"78a865ee-3cce-4990-90cc-521595d38449\") "
Sep 11 12:41:51.948506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.948484    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78a865ee-3cce-4990-90cc-521595d38449" (UID: "78a865ee-3cce-4990-90cc-521595d38449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:41:51.950230 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:51.950210    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s" (OuterVolumeSpecName: "kube-api-access-mkw2s") pod "78a865ee-3cce-4990-90cc-521595d38449" (UID: "78a865ee-3cce-4990-90cc-521595d38449"). InnerVolumeSpecName "kube-api-access-mkw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:52.049879 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.049840    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkw2s\" (UniqueName: \"kubernetes.io/projected/78a865ee-3cce-4990-90cc-521595d38449-kube-api-access-mkw2s\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:52.049879 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.049874    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a865ee-3cce-4990-90cc-521595d38449-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:41:52.409790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.409707    2575 generic.go:356] "Generic (PLEG): container finished" podID="78a865ee-3cce-4990-90cc-521595d38449" containerID="d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6" exitCode=0
Sep 11 12:41:52.409790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.409755    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"
Sep 11 12:41:52.409790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.409758    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" event={"ID":"78a865ee-3cce-4990-90cc-521595d38449","Type":"ContainerDied","Data":"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"}
Sep 11 12:41:52.410018 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.409809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm" event={"ID":"78a865ee-3cce-4990-90cc-521595d38449","Type":"ContainerDied","Data":"56037eda38c0df0612c1efc7bb23aa26b1160d1bdfd74238e2160a5074e4754f"}
Sep 11 12:41:52.410018 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.409839    2575 scope.go:117] "RemoveContainer" containerID="d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"
Sep 11 12:41:52.419463 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.419444    2575 scope.go:117] "RemoveContainer" containerID="d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"
Sep 11 12:41:52.419704 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:41:52.419685    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6\": container with ID starting with d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6 not found: ID does not exist" containerID="d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"
Sep 11 12:41:52.419756 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.419713    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6"} err="failed to get container status \"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6\": rpc error: code = NotFound desc = could not find container \"d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6\": container with ID starting with d358cabdf27bcfb70798a837664e6b4855efff5f77be74d6bcba9f62ba8f8bb6 not found: ID does not exist"
Sep 11 12:41:52.432420 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.432394    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:52.438106 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:52.438083    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-47131dfa-6bf9d68c4b-stlcm"]
Sep 11 12:41:53.629768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:53.629731    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a865ee-3cce-4990-90cc-521595d38449" path="/var/lib/kubelet/pods/78a865ee-3cce-4990-90cc-521595d38449/volumes"
Sep 11 12:41:56.879042 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.879002    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:41:56.879479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.879370    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a865ee-3cce-4990-90cc-521595d38449" containerName="payload-pre-processing"
Sep 11 12:41:56.879479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.879383    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a865ee-3cce-4990-90cc-521595d38449" containerName="payload-pre-processing"
Sep 11 12:41:56.879479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.879456    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a865ee-3cce-4990-90cc-521595d38449" containerName="payload-pre-processing"
Sep 11 12:41:56.882714 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.882694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:56.885541 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.885521    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bdb1ecfc\""
Sep 11 12:41:56.885699 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.885680    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bdb1ecfc-dockercfg-p2b84\""
Sep 11 12:41:56.906061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.906039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:41:56.992813 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.992769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:56.992988 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:56.992893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqwm\" (UniqueName: \"kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.093504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.093465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.093656 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.093554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqwm\" (UniqueName: \"kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.094234 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.094212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.102476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.102445    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqwm\" (UniqueName: \"kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm\") pod \"payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.192971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.192941    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:57.464693 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:57.464668    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:41:57.466830 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:41:57.466803    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571ed107_5268_41d6_b82e_ea9ac8cef054.slice/crio-3bded0c3946c4ac92688c15462f69d664dd0c5bab03a63a8ac4f962e4e340046 WatchSource:0}: Error finding container 3bded0c3946c4ac92688c15462f69d664dd0c5bab03a63a8ac4f962e4e340046: Status 404 returned error can't find the container with id 3bded0c3946c4ac92688c15462f69d664dd0c5bab03a63a8ac4f962e4e340046
Sep 11 12:41:58.445122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:58.445083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" event={"ID":"571ed107-5268-41d6-b82e-ea9ac8cef054","Type":"ContainerStarted","Data":"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"}
Sep 11 12:41:58.445634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:58.445169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" event={"ID":"571ed107-5268-41d6-b82e-ea9ac8cef054","Type":"ContainerStarted","Data":"3bded0c3946c4ac92688c15462f69d664dd0c5bab03a63a8ac4f962e4e340046"}
Sep 11 12:41:58.445634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:58.445202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:41:58.463347 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:41:58.463270    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" podStartSLOduration=2.463250714 podStartE2EDuration="2.463250714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:41:58.46120638 +0000 UTC m=+901.302957524" watchObservedRunningTime="2026-09-11 12:41:58.463250714 +0000 UTC m=+901.305001840"
Sep 11 12:42:03.867565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:03.867531    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:42:03.872208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:03.872189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:03.874718 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:03.874695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-50eee0c3-kserve-self-signed-certs\""
Sep 11 12:42:03.879654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:03.879567    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.059732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.059793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.059913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b5q\" (UniqueName: \"kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.059953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.059988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.063358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.060022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161399 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b5q\" (UniqueName: \"kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161399 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161399 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161600 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161656 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161656 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161842 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.161918 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.161899    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.164007 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.163985    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.164168 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.164151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.173282 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.173261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b5q\" (UniqueName: \"kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q\") pod \"e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") " pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.182205 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.182182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:04.456248 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.456208    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:42:04.458562 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:42:04.458531    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod516fa016_322f_4e30_8a44_a90736eb0254.slice/crio-3d7ccf252ecfbbfd613c7e5e5a0813ffa6deddea5bc599cbabe235a6263aaa66 WatchSource:0}: Error finding container 3d7ccf252ecfbbfd613c7e5e5a0813ffa6deddea5bc599cbabe235a6263aaa66: Status 404 returned error can't find the container with id 3d7ccf252ecfbbfd613c7e5e5a0813ffa6deddea5bc599cbabe235a6263aaa66
Sep 11 12:42:04.474065 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:04.474041    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" event={"ID":"516fa016-322f-4e30-8a44-a90736eb0254","Type":"ContainerStarted","Data":"3d7ccf252ecfbbfd613c7e5e5a0813ffa6deddea5bc599cbabe235a6263aaa66"}
Sep 11 12:42:05.479883 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:05.479847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" event={"ID":"516fa016-322f-4e30-8a44-a90736eb0254","Type":"ContainerStarted","Data":"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"}
Sep 11 12:42:05.499817 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:05.499766    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" podStartSLOduration=2.499751887 podStartE2EDuration="2.499751887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:05.497393102 +0000 UTC m=+908.339144229" watchObservedRunningTime="2026-09-11 12:42:05.499751887 +0000 UTC m=+908.341503013"
Sep 11 12:42:09.453000 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:09.452966    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:42:14.182743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:14.182707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:14.183117 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:14.182767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:14.195511 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:14.195488    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:14.545265 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:14.545231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:42:20.280986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.280946    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:42:20.284902 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.284877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.288837 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.288812    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9878a9\"/\"default-dockercfg-xxkc2\""
Sep 11 12:42:20.289037 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.288819    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9878a9\"/\"kube-root-ca.crt\""
Sep 11 12:42:20.289165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.288841    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9878a9\"/\"openshift-service-ca.crt\""
Sep 11 12:42:20.289165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.288856    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9878a9\"/\"autores-model-9878a9-kserve-self-signed-certs\""
Sep 11 12:42:20.294436 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.294414    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:42:20.410528 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.410731 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7sf\" (UniqueName: \"kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.410731 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.410731 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.411035 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.411035 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.410763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511462 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511462 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7sf\" (UniqueName: \"kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511888 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.511954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.512004 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.511984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.513897 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.513875    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.514070 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.514053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.519926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.519896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7sf\" (UniqueName: \"kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf\") pod \"autores-model-9878a9-kserve-696958459f-gb6nb\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") " pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.595594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.595518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:20.874928 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:20.874904    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:42:20.877350 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:42:20.877303    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765f61e4_e6c9_4293_a164_e6e7e9e5e844.slice/crio-3b528e5a7f38f5aabff1848aaa668ec7927f96bbd061722dbb49f3435d51cbdc WatchSource:0}: Error finding container 3b528e5a7f38f5aabff1848aaa668ec7927f96bbd061722dbb49f3435d51cbdc: Status 404 returned error can't find the container with id 3b528e5a7f38f5aabff1848aaa668ec7927f96bbd061722dbb49f3435d51cbdc
Sep 11 12:42:21.569345 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:21.569288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" event={"ID":"765f61e4-e6c9-4293-a164-e6e7e9e5e844","Type":"ContainerStarted","Data":"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"}
Sep 11 12:42:21.569345 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:21.569341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" event={"ID":"765f61e4-e6c9-4293-a164-e6e7e9e5e844","Type":"ContainerStarted","Data":"3b528e5a7f38f5aabff1848aaa668ec7927f96bbd061722dbb49f3435d51cbdc"}
Sep 11 12:42:21.590045 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:21.589983    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" podStartSLOduration=1.589969349 podStartE2EDuration="1.589969349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:21.588483133 +0000 UTC m=+924.430234284" watchObservedRunningTime="2026-09-11 12:42:21.589969349 +0000 UTC m=+924.431720475"
Sep 11 12:42:26.871803 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:26.871766    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:42:26.872772 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:26.872638    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" podUID="571ed107-5268-41d6-b82e-ea9ac8cef054" containerName="payload-pre-processing" containerID="cri-o://34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe" gracePeriod=30
Sep 11 12:42:27.122429 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.122366    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:42:27.278584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.278546    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume\") pod \"571ed107-5268-41d6-b82e-ea9ac8cef054\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") "
Sep 11 12:42:27.278768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.278610    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqqwm\" (UniqueName: \"kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm\") pod \"571ed107-5268-41d6-b82e-ea9ac8cef054\" (UID: \"571ed107-5268-41d6-b82e-ea9ac8cef054\") "
Sep 11 12:42:27.278898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.278873    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "571ed107-5268-41d6-b82e-ea9ac8cef054" (UID: "571ed107-5268-41d6-b82e-ea9ac8cef054"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:42:27.280622 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.280601    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm" (OuterVolumeSpecName: "kube-api-access-lqqwm") pod "571ed107-5268-41d6-b82e-ea9ac8cef054" (UID: "571ed107-5268-41d6-b82e-ea9ac8cef054"). InnerVolumeSpecName "kube-api-access-lqqwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:42:27.379481 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.379405    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571ed107-5268-41d6-b82e-ea9ac8cef054-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:42:27.379481 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.379434    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqwm\" (UniqueName: \"kubernetes.io/projected/571ed107-5268-41d6-b82e-ea9ac8cef054-kube-api-access-lqqwm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:42:27.598777 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.598744    2575 generic.go:356] "Generic (PLEG): container finished" podID="571ed107-5268-41d6-b82e-ea9ac8cef054" containerID="34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe" exitCode=0
Sep 11 12:42:27.598945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.598791    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"
Sep 11 12:42:27.598945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.598832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" event={"ID":"571ed107-5268-41d6-b82e-ea9ac8cef054","Type":"ContainerDied","Data":"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"}
Sep 11 12:42:27.598945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.598888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb" event={"ID":"571ed107-5268-41d6-b82e-ea9ac8cef054","Type":"ContainerDied","Data":"3bded0c3946c4ac92688c15462f69d664dd0c5bab03a63a8ac4f962e4e340046"}
Sep 11 12:42:27.598945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.598910    2575 scope.go:117] "RemoveContainer" containerID="34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"
Sep 11 12:42:27.607627 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.607610    2575 scope.go:117] "RemoveContainer" containerID="34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"
Sep 11 12:42:27.607890 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:42:27.607872    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe\": container with ID starting with 34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe not found: ID does not exist" containerID="34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"
Sep 11 12:42:27.607944 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.607900    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe"} err="failed to get container status \"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe\": rpc error: code = NotFound desc = could not find container \"34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe\": container with ID starting with 34b264de7d4c7f8dc1f40712efd1efb2875f65e0456edcb7b39ee1b0665dc9fe not found: ID does not exist"
Sep 11 12:42:27.620449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.620424    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:42:27.631101 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:27.631057    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdb1ecfc-699c7f4fd-p7jrb"]
Sep 11 12:42:29.630091 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:29.630048    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="571ed107-5268-41d6-b82e-ea9ac8cef054" path="/var/lib/kubelet/pods/571ed107-5268-41d6-b82e-ea9ac8cef054/volumes"
Sep 11 12:42:30.596124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:30.596087    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:30.596302 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:30.596143    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:30.608714 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:30.608686    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:30.623693 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:30.623669    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:42:33.124006 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.123979    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:42:33.124407 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.124355    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571ed107-5268-41d6-b82e-ea9ac8cef054" containerName="payload-pre-processing"
Sep 11 12:42:33.124407 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.124366    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="571ed107-5268-41d6-b82e-ea9ac8cef054" containerName="payload-pre-processing"
Sep 11 12:42:33.124491 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.124430    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="571ed107-5268-41d6-b82e-ea9ac8cef054" containerName="payload-pre-processing"
Sep 11 12:42:33.128811 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.128793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.131777 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.131755    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d76886ba-dockercfg-49n9v\""
Sep 11 12:42:33.131876 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.131793    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d76886ba\""
Sep 11 12:42:33.139376 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.139355    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:42:33.232707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.232673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hlv\" (UniqueName: \"kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.232870 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.232723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.334241 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.334207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hlv\" (UniqueName: \"kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.334426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.334249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.334963 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.334945    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.342496 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.342472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hlv\" (UniqueName: \"kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv\") pod \"payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") " pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.440368 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.440341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:33.716012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:33.715986    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:42:33.718071 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:42:33.718040    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc321b1ce_fb11_4a79_a100_fb5a70c05039.slice/crio-660b2a2f044935f4ea9d60ee07f3061b72b76685286079e86e742d1da01eec05 WatchSource:0}: Error finding container 660b2a2f044935f4ea9d60ee07f3061b72b76685286079e86e742d1da01eec05: Status 404 returned error can't find the container with id 660b2a2f044935f4ea9d60ee07f3061b72b76685286079e86e742d1da01eec05
Sep 11 12:42:34.636928 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:34.636895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" event={"ID":"c321b1ce-fb11-4a79-a100-fb5a70c05039","Type":"ContainerStarted","Data":"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"}
Sep 11 12:42:34.637295 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:34.636939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" event={"ID":"c321b1ce-fb11-4a79-a100-fb5a70c05039","Type":"ContainerStarted","Data":"660b2a2f044935f4ea9d60ee07f3061b72b76685286079e86e742d1da01eec05"}
Sep 11 12:42:34.637295 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:34.636980    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:34.657283 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:34.657240    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" podStartSLOduration=1.657226772 podStartE2EDuration="1.657226772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:34.654489455 +0000 UTC m=+937.496240600" watchObservedRunningTime="2026-09-11 12:42:34.657226772 +0000 UTC m=+937.498977898"
Sep 11 12:42:45.643385 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:45.643355    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:42:47.001648 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.001612    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:42:47.005393 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.005372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.009889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.009865    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edce0b\"/\"kube-root-ca.crt\""
Sep 11 12:42:47.009889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.009874    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edce0b\"/\"test-model-edce0b-kserve-self-signed-certs\""
Sep 11 12:42:47.009889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.009875    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edce0b\"/\"default-dockercfg-5gjf2\""
Sep 11 12:42:47.010124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.009955    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edce0b\"/\"openshift-service-ca.crt\""
Sep 11 12:42:47.016781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.016757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:42:47.052005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.051976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcx4d\" (UniqueName: \"kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.052095 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.052018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.052095 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.052035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.052095 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.052056    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.052203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.052095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.052203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.052181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.152709 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.152819 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcx4d\" (UniqueName: \"kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.152819 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.152819 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.152819 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.153009 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.152850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.153209 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.153190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.153294 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.153221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.153294 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.153278    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.155011 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.154980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.155336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.155294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.161585 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.161566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcx4d\" (UniqueName: \"kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d\") pod \"test-model-edce0b-kserve-549b9ff46c-c5k9s\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") " pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.314866 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.314788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:47.585083 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.585060    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:42:47.586926 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:42:47.586893    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ad1aa6a_3846_476f_971e_cfdd23cc91a4.slice/crio-1b1473ac793f07b498a2caf0a4d94b5de254d3aac8544aedf2078971d82db894 WatchSource:0}: Error finding container 1b1473ac793f07b498a2caf0a4d94b5de254d3aac8544aedf2078971d82db894: Status 404 returned error can't find the container with id 1b1473ac793f07b498a2caf0a4d94b5de254d3aac8544aedf2078971d82db894
Sep 11 12:42:47.699205 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.699177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" event={"ID":"3ad1aa6a-3846-476f-971e-cfdd23cc91a4","Type":"ContainerStarted","Data":"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"}
Sep 11 12:42:47.699335 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.699214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" event={"ID":"3ad1aa6a-3846-476f-971e-cfdd23cc91a4","Type":"ContainerStarted","Data":"1b1473ac793f07b498a2caf0a4d94b5de254d3aac8544aedf2078971d82db894"}
Sep 11 12:42:47.718013 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:47.717962    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" podStartSLOduration=1.717945513 podStartE2EDuration="1.717945513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:47.716716136 +0000 UTC m=+950.558467281" watchObservedRunningTime="2026-09-11 12:42:47.717945513 +0000 UTC m=+950.559696639"
Sep 11 12:42:50.081877 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:50.081834    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:42:50.983871 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:50.983833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:42:57.315248 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:57.315210    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:57.315672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:57.315290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:57.327670 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:57.327646    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:42:57.756638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:42:57.756609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:43:01.808504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:01.808472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:43:01.809077 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:01.808673    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" podUID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" containerName="payload-processing" containerID="cri-o://650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c" gracePeriod=30
Sep 11 12:43:01.819224 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:01.819183    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:43:01.819652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:01.819474    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" podUID="c8955d5e-a652-45aa-b836-63e6a29106b1" containerName="payload-pre-processing" containerID="cri-o://9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c" gracePeriod=30
Sep 11 12:43:02.086130 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.086106    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:43:02.088707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.088686    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:43:02.197378 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197319    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpqvp\" (UniqueName: \"kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp\") pod \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") "
Sep 11 12:43:02.197378 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197384    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume\") pod \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\" (UID: \"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e\") "
Sep 11 12:43:02.197628 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l59k\" (UniqueName: \"kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k\") pod \"c8955d5e-a652-45aa-b836-63e6a29106b1\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") "
Sep 11 12:43:02.197628 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197496    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume\") pod \"c8955d5e-a652-45aa-b836-63e6a29106b1\" (UID: \"c8955d5e-a652-45aa-b836-63e6a29106b1\") "
Sep 11 12:43:02.197860 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" (UID: "6c1d6cb5-51e2-4824-8e69-61c0496a2e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:43:02.197972 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.197906    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8955d5e-a652-45aa-b836-63e6a29106b1" (UID: "c8955d5e-a652-45aa-b836-63e6a29106b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:43:02.199639 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.199609    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp" (OuterVolumeSpecName: "kube-api-access-hpqvp") pod "6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" (UID: "6c1d6cb5-51e2-4824-8e69-61c0496a2e3e"). InnerVolumeSpecName "kube-api-access-hpqvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:02.199741 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.199641    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k" (OuterVolumeSpecName: "kube-api-access-9l59k") pod "c8955d5e-a652-45aa-b836-63e6a29106b1" (UID: "c8955d5e-a652-45aa-b836-63e6a29106b1"). InnerVolumeSpecName "kube-api-access-9l59k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:02.298910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.298882    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:02.298910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.298907    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l59k\" (UniqueName: \"kubernetes.io/projected/c8955d5e-a652-45aa-b836-63e6a29106b1-kube-api-access-9l59k\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:02.299062 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.298917    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8955d5e-a652-45aa-b836-63e6a29106b1-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:02.299062 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.298925    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqvp\" (UniqueName: \"kubernetes.io/projected/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e-kube-api-access-hpqvp\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:02.770368 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.770313    2575 generic.go:356] "Generic (PLEG): container finished" podID="c8955d5e-a652-45aa-b836-63e6a29106b1" containerID="9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c" exitCode=0
Sep 11 12:43:02.770565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.770382    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"
Sep 11 12:43:02.770565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.770410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" event={"ID":"c8955d5e-a652-45aa-b836-63e6a29106b1","Type":"ContainerDied","Data":"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"}
Sep 11 12:43:02.770565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.770463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc" event={"ID":"c8955d5e-a652-45aa-b836-63e6a29106b1","Type":"ContainerDied","Data":"7e08fcde9adfa84e75af78ca976d52792cb68b7d094fa5fabf29909b4b2512b9"}
Sep 11 12:43:02.770565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.770494    2575 scope.go:117] "RemoveContainer" containerID="9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"
Sep 11 12:43:02.771845 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.771818    2575 generic.go:356] "Generic (PLEG): container finished" podID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" containerID="650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c" exitCode=0
Sep 11 12:43:02.771954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.771861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" event={"ID":"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e","Type":"ContainerDied","Data":"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"}
Sep 11 12:43:02.771954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.771902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8" event={"ID":"6c1d6cb5-51e2-4824-8e69-61c0496a2e3e","Type":"ContainerDied","Data":"bda87a7aecd531f0e4438365a36eb185c8442c9417159e4660299c30a644b037"}
Sep 11 12:43:02.771954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.771864    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"
Sep 11 12:43:02.779945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.779930    2575 scope.go:117] "RemoveContainer" containerID="9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"
Sep 11 12:43:02.780235 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:43:02.780212    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c\": container with ID starting with 9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c not found: ID does not exist" containerID="9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"
Sep 11 12:43:02.780304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.780243    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c"} err="failed to get container status \"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c\": rpc error: code = NotFound desc = could not find container \"9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c\": container with ID starting with 9e7e20a07bfbadff50661a4e2a53b5723f7577e718a20e9f010a09027830cf0c not found: ID does not exist"
Sep 11 12:43:02.780304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.780260    2575 scope.go:117] "RemoveContainer" containerID="650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"
Sep 11 12:43:02.787867 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.787851    2575 scope.go:117] "RemoveContainer" containerID="650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"
Sep 11 12:43:02.788105 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:43:02.788088    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c\": container with ID starting with 650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c not found: ID does not exist" containerID="650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"
Sep 11 12:43:02.788141 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.788112    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c"} err="failed to get container status \"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c\": rpc error: code = NotFound desc = could not find container \"650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c\": container with ID starting with 650921ea0de4add8afe1562f535e99b7c405e56be8f8ffd0b4eda65f5a15e54c not found: ID does not exist"
Sep 11 12:43:02.804617 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.804594    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:43:02.809124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.809101    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9878a9-58ffbfbffc-crdm8"]
Sep 11 12:43:02.819432 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.819412    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:43:02.824795 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:02.824772    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9878a9-97f8d446-sldnc"]
Sep 11 12:43:03.338438 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.338407    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:43:03.338666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.338619    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" podUID="c321b1ce-fb11-4a79-a100-fb5a70c05039" containerName="payload-processing" containerID="cri-o://58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104" gracePeriod=30
Sep 11 12:43:03.582673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.582651    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:43:03.633371 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.633266    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" path="/var/lib/kubelet/pods/6c1d6cb5-51e2-4824-8e69-61c0496a2e3e/volumes"
Sep 11 12:43:03.633818 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.633799    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8955d5e-a652-45aa-b836-63e6a29106b1" path="/var/lib/kubelet/pods/c8955d5e-a652-45aa-b836-63e6a29106b1/volumes"
Sep 11 12:43:03.711501 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.711466    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8hlv\" (UniqueName: \"kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv\") pod \"c321b1ce-fb11-4a79-a100-fb5a70c05039\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") "
Sep 11 12:43:03.711679 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.711534    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume\") pod \"c321b1ce-fb11-4a79-a100-fb5a70c05039\" (UID: \"c321b1ce-fb11-4a79-a100-fb5a70c05039\") "
Sep 11 12:43:03.711931 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.711906    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c321b1ce-fb11-4a79-a100-fb5a70c05039" (UID: "c321b1ce-fb11-4a79-a100-fb5a70c05039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:43:03.713645 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.713621    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv" (OuterVolumeSpecName: "kube-api-access-j8hlv") pod "c321b1ce-fb11-4a79-a100-fb5a70c05039" (UID: "c321b1ce-fb11-4a79-a100-fb5a70c05039"). InnerVolumeSpecName "kube-api-access-j8hlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:03.779447 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.779412    2575 generic.go:356] "Generic (PLEG): container finished" podID="c321b1ce-fb11-4a79-a100-fb5a70c05039" containerID="58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104" exitCode=0
Sep 11 12:43:03.779594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.779471    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"
Sep 11 12:43:03.779594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.779499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" event={"ID":"c321b1ce-fb11-4a79-a100-fb5a70c05039","Type":"ContainerDied","Data":"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"}
Sep 11 12:43:03.779594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.779585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp" event={"ID":"c321b1ce-fb11-4a79-a100-fb5a70c05039","Type":"ContainerDied","Data":"660b2a2f044935f4ea9d60ee07f3061b72b76685286079e86e742d1da01eec05"}
Sep 11 12:43:03.779781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.779615    2575 scope.go:117] "RemoveContainer" containerID="58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"
Sep 11 12:43:03.788175 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.788155    2575 scope.go:117] "RemoveContainer" containerID="58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"
Sep 11 12:43:03.788443 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:43:03.788421    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104\": container with ID starting with 58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104 not found: ID does not exist" containerID="58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"
Sep 11 12:43:03.788512 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.788450    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104"} err="failed to get container status \"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104\": rpc error: code = NotFound desc = could not find container \"58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104\": container with ID starting with 58740a1f64d932789dbad9bbcf1e2be3b34a0946c039eb1b460e2d692c900104 not found: ID does not exist"
Sep 11 12:43:03.800586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.800565    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:43:03.804464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.804444    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d76886ba-85796ccb6f-c27kp"]
Sep 11 12:43:03.813154 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.813139    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hlv\" (UniqueName: \"kubernetes.io/projected/c321b1ce-fb11-4a79-a100-fb5a70c05039-kube-api-access-j8hlv\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:03.813476 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:03.813157    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c321b1ce-fb11-4a79-a100-fb5a70c05039-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.101781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.101741    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" podUID="516fa016-322f-4e30-8a44-a90736eb0254" containerName="main" containerID="cri-o://aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98" gracePeriod=45
Sep 11 12:43:05.355794 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.355740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:43:05.428195 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428163    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85b5q\" (UniqueName: \"kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428209    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428236    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428281    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428296    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428389 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428357    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home\") pod \"516fa016-322f-4e30-8a44-a90736eb0254\" (UID: \"516fa016-322f-4e30-8a44-a90736eb0254\") "
Sep 11 12:43:05.428646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428547    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:05.428699 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428647    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache" (OuterVolumeSpecName: "model-cache") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:05.428699 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428659    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home" (OuterVolumeSpecName: "home") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:05.428699 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.428676    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.430505 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.430484    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:43:05.430565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.430482    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm" (OuterVolumeSpecName: "dshm") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:05.430565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.430490    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q" (OuterVolumeSpecName: "kube-api-access-85b5q") pod "516fa016-322f-4e30-8a44-a90736eb0254" (UID: "516fa016-322f-4e30-8a44-a90736eb0254"). InnerVolumeSpecName "kube-api-access-85b5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:05.529344 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.529292    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.529344 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.529321    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516fa016-322f-4e30-8a44-a90736eb0254-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.529344 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.529350    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.529541 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.529358    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b5q\" (UniqueName: \"kubernetes.io/projected/516fa016-322f-4e30-8a44-a90736eb0254-kube-api-access-85b5q\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.529541 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.529368    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516fa016-322f-4e30-8a44-a90736eb0254-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:05.630717 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.630645    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c321b1ce-fb11-4a79-a100-fb5a70c05039" path="/var/lib/kubelet/pods/c321b1ce-fb11-4a79-a100-fb5a70c05039/volumes"
Sep 11 12:43:05.793957 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.793921    2575 generic.go:356] "Generic (PLEG): container finished" podID="516fa016-322f-4e30-8a44-a90736eb0254" containerID="aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98" exitCode=0
Sep 11 12:43:05.793957 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.793957    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"
Sep 11 12:43:05.794182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.794017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" event={"ID":"516fa016-322f-4e30-8a44-a90736eb0254","Type":"ContainerDied","Data":"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"}
Sep 11 12:43:05.794182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.794055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh" event={"ID":"516fa016-322f-4e30-8a44-a90736eb0254","Type":"ContainerDied","Data":"3d7ccf252ecfbbfd613c7e5e5a0813ffa6deddea5bc599cbabe235a6263aaa66"}
Sep 11 12:43:05.794182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.794074    2575 scope.go:117] "RemoveContainer" containerID="aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"
Sep 11 12:43:05.804351 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.804308    2575 scope.go:117] "RemoveContainer" containerID="aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"
Sep 11 12:43:05.804616 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:43:05.804596    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98\": container with ID starting with aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98 not found: ID does not exist" containerID="aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"
Sep 11 12:43:05.804692 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.804622    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98"} err="failed to get container status \"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98\": rpc error: code = NotFound desc = could not find container \"aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98\": container with ID starting with aaa0e1302f430322126d845e0c21c2e93014be3b5f72a7cd877d26de03be5d98 not found: ID does not exist"
Sep 11 12:43:05.812805 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.812783    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:43:05.817041 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:05.817022    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-50eee0c3-kserve-7bf5ccfbbc-v5jmh"]
Sep 11 12:43:06.003052 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.002990    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" podUID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" containerName="main" containerID="cri-o://363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900" gracePeriod=45
Sep 11 12:43:06.255340 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.255267    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:43:06.335983 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.335954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.335998    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q7sf\" (UniqueName: \"kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336018    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336057    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336194 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336099    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm\") pod \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\" (UID: \"765f61e4-e6c9-4293-a164-e6e7e9e5e844\") "
Sep 11 12:43:06.336480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336251    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache" (OuterVolumeSpecName: "model-cache") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:06.336480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336281    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:06.336480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336349    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home" (OuterVolumeSpecName: "home") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:06.336480 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.336398    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.338199 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.338167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:43:06.338199 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.338183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm" (OuterVolumeSpecName: "dshm") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:06.338358 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.338177    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf" (OuterVolumeSpecName: "kube-api-access-9q7sf") pod "765f61e4-e6c9-4293-a164-e6e7e9e5e844" (UID: "765f61e4-e6c9-4293-a164-e6e7e9e5e844"). InnerVolumeSpecName "kube-api-access-9q7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:06.437804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.437772    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.437804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.437798    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.437804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.437806    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.438017 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.437815    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7sf\" (UniqueName: \"kubernetes.io/projected/765f61e4-e6c9-4293-a164-e6e7e9e5e844-kube-api-access-9q7sf\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.438017 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.437825    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765f61e4-e6c9-4293-a164-e6e7e9e5e844-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:43:06.800670 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.800634    2575 generic.go:356] "Generic (PLEG): container finished" podID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" containerID="363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900" exitCode=0
Sep 11 12:43:06.800814 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.800679    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"
Sep 11 12:43:06.800814 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.800715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" event={"ID":"765f61e4-e6c9-4293-a164-e6e7e9e5e844","Type":"ContainerDied","Data":"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"}
Sep 11 12:43:06.800814 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.800757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb" event={"ID":"765f61e4-e6c9-4293-a164-e6e7e9e5e844","Type":"ContainerDied","Data":"3b528e5a7f38f5aabff1848aaa668ec7927f96bbd061722dbb49f3435d51cbdc"}
Sep 11 12:43:06.800814 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.800774    2575 scope.go:117] "RemoveContainer" containerID="363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"
Sep 11 12:43:06.809281 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.809267    2575 scope.go:117] "RemoveContainer" containerID="363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"
Sep 11 12:43:06.809586 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:43:06.809570    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900\": container with ID starting with 363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900 not found: ID does not exist" containerID="363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"
Sep 11 12:43:06.809638 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.809594    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900"} err="failed to get container status \"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900\": rpc error: code = NotFound desc = could not find container \"363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900\": container with ID starting with 363fab5955064138e7f349130f219a47de0143f1220d4456cbd11a888b8ce900 not found: ID does not exist"
Sep 11 12:43:06.821930 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.821902    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:43:06.823785 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:06.823760    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-9878a9/autores-model-9878a9-kserve-696958459f-gb6nb"]
Sep 11 12:43:07.635873 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:07.635837    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="516fa016-322f-4e30-8a44-a90736eb0254" path="/var/lib/kubelet/pods/516fa016-322f-4e30-8a44-a90736eb0254/volumes"
Sep 11 12:43:07.636236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:07.636222    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" path="/var/lib/kubelet/pods/765f61e4-e6c9-4293-a164-e6e7e9e5e844/volumes"
Sep 11 12:43:13.987754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.987717    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988060    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" containerName="main"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" containerName="main"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988082    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="516fa016-322f-4e30-8a44-a90736eb0254" containerName="main"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988088    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="516fa016-322f-4e30-8a44-a90736eb0254" containerName="main"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988102    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8955d5e-a652-45aa-b836-63e6a29106b1" containerName="payload-pre-processing"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988109    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8955d5e-a652-45aa-b836-63e6a29106b1" containerName="payload-pre-processing"
Sep 11 12:43:13.988116 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988120    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c321b1ce-fb11-4a79-a100-fb5a70c05039" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988126    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c321b1ce-fb11-4a79-a100-fb5a70c05039" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988134    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988139    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988193    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="765f61e4-e6c9-4293-a164-e6e7e9e5e844" containerName="main"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988200    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="516fa016-322f-4e30-8a44-a90736eb0254" containerName="main"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988206    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c321b1ce-fb11-4a79-a100-fb5a70c05039" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988214    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1d6cb5-51e2-4824-8e69-61c0496a2e3e" containerName="payload-processing"
Sep 11 12:43:13.988336 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.988219    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8955d5e-a652-45aa-b836-63e6a29106b1" containerName="payload-pre-processing"
Sep 11 12:43:13.992430 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.992414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:13.995281 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.995262    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b8dbb2-dockercfg-7kh5g\""
Sep 11 12:43:13.995413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:13.995350    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b8dbb2\""
Sep 11 12:43:14.002500 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.002476    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:43:14.021431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.021393    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:43:14.029872 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.029839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.037153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.037123    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:43:14.103869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.103831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.104029 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.103876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85c7t\" (UniqueName: \"kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.205028 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.204994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.205028 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.205034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85c7t\" (UniqueName: \"kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.205237 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.205065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28cz\" (UniqueName: \"kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.205237 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.205128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.205763 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.205744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.213929 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.213897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85c7t\" (UniqueName: \"kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t\") pod \"payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.303713 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.303609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.306024 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.305990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28cz\" (UniqueName: \"kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.306085 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.306061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.306855 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.306831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.314611 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.314591    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28cz\" (UniqueName: \"kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz\") pod \"payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.341562 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.341522    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.576665 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.576637    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:43:14.578691 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:43:14.578650    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf588d7d2_a21b_42ce_a400_493cbf7e023a.slice/crio-6ca5df5ec9ffc70481c06e98a545a2ce940d21f3b6746cf2cc4fc6e57efbaa5e WatchSource:0}: Error finding container 6ca5df5ec9ffc70481c06e98a545a2ce940d21f3b6746cf2cc4fc6e57efbaa5e: Status 404 returned error can't find the container with id 6ca5df5ec9ffc70481c06e98a545a2ce940d21f3b6746cf2cc4fc6e57efbaa5e
Sep 11 12:43:14.624475 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.624437    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:43:14.636312 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:43:14.636286    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f6f4fa_5d03_44d0_968f_da54d04bfa5e.slice/crio-05d35b993dade2cd22aae55424764a03ccbaafd51ca4a2e0a002df59f9d71a34 WatchSource:0}: Error finding container 05d35b993dade2cd22aae55424764a03ccbaafd51ca4a2e0a002df59f9d71a34: Status 404 returned error can't find the container with id 05d35b993dade2cd22aae55424764a03ccbaafd51ca4a2e0a002df59f9d71a34
Sep 11 12:43:14.842828 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.842732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" event={"ID":"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e","Type":"ContainerStarted","Data":"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"}
Sep 11 12:43:14.842828 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.842789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" event={"ID":"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e","Type":"ContainerStarted","Data":"05d35b993dade2cd22aae55424764a03ccbaafd51ca4a2e0a002df59f9d71a34"}
Sep 11 12:43:14.842828 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.842827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:14.844436 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.844412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" event={"ID":"f588d7d2-a21b-42ce-a400-493cbf7e023a","Type":"ContainerStarted","Data":"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"}
Sep 11 12:43:14.844555 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.844451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" event={"ID":"f588d7d2-a21b-42ce-a400-493cbf7e023a","Type":"ContainerStarted","Data":"6ca5df5ec9ffc70481c06e98a545a2ce940d21f3b6746cf2cc4fc6e57efbaa5e"}
Sep 11 12:43:14.844555 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.844541    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:14.861924 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.861882    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" podStartSLOduration=1.861868404 podStartE2EDuration="1.861868404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:14.860016592 +0000 UTC m=+977.701767750" watchObservedRunningTime="2026-09-11 12:43:14.861868404 +0000 UTC m=+977.703619529"
Sep 11 12:43:14.878375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:14.878316    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" podStartSLOduration=1.878297903 podStartE2EDuration="1.878297903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:14.876272182 +0000 UTC m=+977.718023331" watchObservedRunningTime="2026-09-11 12:43:14.878297903 +0000 UTC m=+977.720049054"
Sep 11 12:43:25.852960 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:25.852882    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:43:25.852960 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:25.852952    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:43:50.276114 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.276078    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:43:50.279763 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.279746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.284003 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.283974    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f06aba\"/\"test-model-f06aba-kserve-self-signed-certs\""
Sep 11 12:43:50.284003 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.283987    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f06aba\"/\"kube-root-ca.crt\""
Sep 11 12:43:50.284182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.284005    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f06aba\"/\"openshift-service-ca.crt\""
Sep 11 12:43:50.284182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.284029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f06aba\"/\"default-dockercfg-7hjhb\""
Sep 11 12:43:50.289280 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.289258    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:43:50.428523 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.428673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.428673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzzq\" (UniqueName: \"kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.428673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.428673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.428805 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.428711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529590 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529590 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzzq\" (UniqueName: \"kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.529796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.529715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.530252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.530220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.530388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.530251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.530388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.530270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.531968 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.531942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.532159 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.532143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.538162 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.538134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzzq\" (UniqueName: \"kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq\") pod \"test-model-f06aba-kserve-5db56d44b5-8bg5q\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") " pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.590063 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.590034    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:43:50.860213 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:50.860188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:43:50.862348 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:43:50.862290    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod223fe8c7_37fa_47ea_b7f6_d6ca65a8a774.slice/crio-30664f66f3284127f7a8b5b8430e8a1129c19496a9d1108edf526f5f766a0eda WatchSource:0}: Error finding container 30664f66f3284127f7a8b5b8430e8a1129c19496a9d1108edf526f5f766a0eda: Status 404 returned error can't find the container with id 30664f66f3284127f7a8b5b8430e8a1129c19496a9d1108edf526f5f766a0eda
Sep 11 12:43:51.016781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:51.016746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" event={"ID":"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774","Type":"ContainerStarted","Data":"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"}
Sep 11 12:43:51.016922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:51.016793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" event={"ID":"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774","Type":"ContainerStarted","Data":"30664f66f3284127f7a8b5b8430e8a1129c19496a9d1108edf526f5f766a0eda"}
Sep 11 12:43:51.034505 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:43:51.034455    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" podStartSLOduration=1.034437886 podStartE2EDuration="1.034437886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:51.033127684 +0000 UTC m=+1013.874878830" watchObservedRunningTime="2026-09-11 12:43:51.034437886 +0000 UTC m=+1013.876189044"
Sep 11 12:44:00.590635 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:00.590595    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:44:00.591004 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:00.590645    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:44:00.602790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:00.602760    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:44:01.077554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:01.077522    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:44:20.863903 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:20.863869    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:44:20.864449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:20.864113    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" podUID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" containerName="payload-processing" containerID="cri-o://cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb" gracePeriod=30
Sep 11 12:44:20.868187 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:20.868141    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:44:20.868444 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:20.868397    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" podUID="f588d7d2-a21b-42ce-a400-493cbf7e023a" containerName="payload-pre-processing" containerID="cri-o://0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637" gracePeriod=30
Sep 11 12:44:21.141045 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.141010    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:44:21.143796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.143778    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:44:21.164802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.164771    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" containerID="cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb" exitCode=0
Sep 11 12:44:21.164922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.164807    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"
Sep 11 12:44:21.164922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.164859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" event={"ID":"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e","Type":"ContainerDied","Data":"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"}
Sep 11 12:44:21.164922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.164918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg" event={"ID":"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e","Type":"ContainerDied","Data":"05d35b993dade2cd22aae55424764a03ccbaafd51ca4a2e0a002df59f9d71a34"}
Sep 11 12:44:21.165089 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.164948    2575 scope.go:117] "RemoveContainer" containerID="cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"
Sep 11 12:44:21.166536 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.166514    2575 generic.go:356] "Generic (PLEG): container finished" podID="f588d7d2-a21b-42ce-a400-493cbf7e023a" containerID="0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637" exitCode=0
Sep 11 12:44:21.166674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.166562    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"
Sep 11 12:44:21.166674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.166599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" event={"ID":"f588d7d2-a21b-42ce-a400-493cbf7e023a","Type":"ContainerDied","Data":"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"}
Sep 11 12:44:21.166674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.166633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr" event={"ID":"f588d7d2-a21b-42ce-a400-493cbf7e023a","Type":"ContainerDied","Data":"6ca5df5ec9ffc70481c06e98a545a2ce940d21f3b6746cf2cc4fc6e57efbaa5e"}
Sep 11 12:44:21.174917 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.174894    2575 scope.go:117] "RemoveContainer" containerID="cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"
Sep 11 12:44:21.175262 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:44:21.175238    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb\": container with ID starting with cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb not found: ID does not exist" containerID="cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"
Sep 11 12:44:21.175401 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.175274    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb"} err="failed to get container status \"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb\": rpc error: code = NotFound desc = could not find container \"cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb\": container with ID starting with cd7c04fdab680c49da7a7b64952bb77f60f6c8f94f0371557dbd226ac434aafb not found: ID does not exist"
Sep 11 12:44:21.175401 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.175299    2575 scope.go:117] "RemoveContainer" containerID="0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"
Sep 11 12:44:21.185705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.185685    2575 scope.go:117] "RemoveContainer" containerID="0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"
Sep 11 12:44:21.185978 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:44:21.185958    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637\": container with ID starting with 0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637 not found: ID does not exist" containerID="0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"
Sep 11 12:44:21.186045 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.185983    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637"} err="failed to get container status \"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637\": rpc error: code = NotFound desc = could not find container \"0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637\": container with ID starting with 0771343ec883c6ca24e464748b49b0ea7e15d2cdc8d00facdc37cc8781ba5637 not found: ID does not exist"
Sep 11 12:44:21.193274 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193255    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k28cz\" (UniqueName: \"kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz\") pod \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") "
Sep 11 12:44:21.193354 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume\") pod \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\" (UID: \"b7f6f4fa-5d03-44d0-968f-da54d04bfa5e\") "
Sep 11 12:44:21.193449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193435    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume\") pod \"f588d7d2-a21b-42ce-a400-493cbf7e023a\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") "
Sep 11 12:44:21.193493 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193481    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85c7t\" (UniqueName: \"kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t\") pod \"f588d7d2-a21b-42ce-a400-493cbf7e023a\" (UID: \"f588d7d2-a21b-42ce-a400-493cbf7e023a\") "
Sep 11 12:44:21.193739 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193711    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" (UID: "b7f6f4fa-5d03-44d0-968f-da54d04bfa5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:44:21.193863 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.193830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f588d7d2-a21b-42ce-a400-493cbf7e023a" (UID: "f588d7d2-a21b-42ce-a400-493cbf7e023a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:44:21.195387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.195359    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz" (OuterVolumeSpecName: "kube-api-access-k28cz") pod "b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" (UID: "b7f6f4fa-5d03-44d0-968f-da54d04bfa5e"). InnerVolumeSpecName "kube-api-access-k28cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:44:21.195652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.195630    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t" (OuterVolumeSpecName: "kube-api-access-85c7t") pod "f588d7d2-a21b-42ce-a400-493cbf7e023a" (UID: "f588d7d2-a21b-42ce-a400-493cbf7e023a"). InnerVolumeSpecName "kube-api-access-85c7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:44:21.294153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.294124    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f588d7d2-a21b-42ce-a400-493cbf7e023a-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:44:21.294153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.294150    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85c7t\" (UniqueName: \"kubernetes.io/projected/f588d7d2-a21b-42ce-a400-493cbf7e023a-kube-api-access-85c7t\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:44:21.294307 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.294165    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28cz\" (UniqueName: \"kubernetes.io/projected/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-kube-api-access-k28cz\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:44:21.294307 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.294177    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:44:21.488405 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.488379    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:44:21.491746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.491722    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b8dbb2-7cf6cc9b77-x5mcg"]
Sep 11 12:44:21.503185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.503162    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:44:21.506670 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.506644    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b8dbb2-6d7f699878-g9phr"]
Sep 11 12:44:21.630257 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.630224    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" path="/var/lib/kubelet/pods/b7f6f4fa-5d03-44d0-968f-da54d04bfa5e/volumes"
Sep 11 12:44:21.630627 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:21.630611    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f588d7d2-a21b-42ce-a400-493cbf7e023a" path="/var/lib/kubelet/pods/f588d7d2-a21b-42ce-a400-493cbf7e023a/volumes"
Sep 11 12:44:43.463020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.462988    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:44:43.463431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463365    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f588d7d2-a21b-42ce-a400-493cbf7e023a" containerName="payload-pre-processing"
Sep 11 12:44:43.463431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463377    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f588d7d2-a21b-42ce-a400-493cbf7e023a" containerName="payload-pre-processing"
Sep 11 12:44:43.463431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463388    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" containerName="payload-processing"
Sep 11 12:44:43.463431 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463394    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" containerName="payload-processing"
Sep 11 12:44:43.463558 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463450    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f588d7d2-a21b-42ce-a400-493cbf7e023a" containerName="payload-pre-processing"
Sep 11 12:44:43.463558 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.463457    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f6f4fa-5d03-44d0-968f-da54d04bfa5e" containerName="payload-processing"
Sep 11 12:44:43.466585 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.466566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.469981 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.469958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-54e055b6\""
Sep 11 12:44:43.469981 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.469973    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-54e055b6-dockercfg-hvtbz\""
Sep 11 12:44:43.478908 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.478885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:44:43.505342 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.505299    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:44:43.508696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.508682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.519563 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.519542    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:44:43.574482 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.574452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.574668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.574500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd7n\" (UniqueName: \"kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.574668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.574594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.574796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.574673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq94n\" (UniqueName: \"kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.675168 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.675137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd7n\" (UniqueName: \"kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.675168 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.675174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.675423 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.675199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq94n\" (UniqueName: \"kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.675423 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.675298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.676142 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.676119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.676217 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.676119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.683533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.683511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq94n\" (UniqueName: \"kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n\") pod \"payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") " pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:43.683666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.683643    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd7n\" (UniqueName: \"kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n\") pod \"payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.776598 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.776524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:43.818204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:43.818172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:44.058735 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.058708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:44:44.061000 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:44:44.060977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf84f24_c4d6_4abe_bf77_2be1c3cb9c90.slice/crio-c3140ab29c37f1e80b30a133a6a40f79ddfaf4ac6998cd425f46044e4d68e517 WatchSource:0}: Error finding container c3140ab29c37f1e80b30a133a6a40f79ddfaf4ac6998cd425f46044e4d68e517: Status 404 returned error can't find the container with id c3140ab29c37f1e80b30a133a6a40f79ddfaf4ac6998cd425f46044e4d68e517
Sep 11 12:44:44.098447 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.098425    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:44:44.099771 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:44:44.099749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fc866_c354_473f_88c1_7a2ca1826435.slice/crio-ff2b189e64ce75111827417424f7cf0a05f01a57e5baf7cfed234d093ec16985 WatchSource:0}: Error finding container ff2b189e64ce75111827417424f7cf0a05f01a57e5baf7cfed234d093ec16985: Status 404 returned error can't find the container with id ff2b189e64ce75111827417424f7cf0a05f01a57e5baf7cfed234d093ec16985
Sep 11 12:44:44.281708 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.281671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" event={"ID":"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90","Type":"ContainerStarted","Data":"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"}
Sep 11 12:44:44.281903 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.281732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" event={"ID":"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90","Type":"ContainerStarted","Data":"c3140ab29c37f1e80b30a133a6a40f79ddfaf4ac6998cd425f46044e4d68e517"}
Sep 11 12:44:44.281903 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.281795    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:44.283407 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.283376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" event={"ID":"a19fc866-c354-473f-88c1-7a2ca1826435","Type":"ContainerStarted","Data":"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"}
Sep 11 12:44:44.283543 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.283422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" event={"ID":"a19fc866-c354-473f-88c1-7a2ca1826435","Type":"ContainerStarted","Data":"ff2b189e64ce75111827417424f7cf0a05f01a57e5baf7cfed234d093ec16985"}
Sep 11 12:44:44.283543 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.283449    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:44:44.301985 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.301935    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" podStartSLOduration=1.301916334 podStartE2EDuration="1.301916334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:44.299377683 +0000 UTC m=+1067.141128834" watchObservedRunningTime="2026-09-11 12:44:44.301916334 +0000 UTC m=+1067.143667463"
Sep 11 12:44:44.317391 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:44.317302    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" podStartSLOduration=1.317290951 podStartE2EDuration="1.317290951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:44.315696918 +0000 UTC m=+1067.157448069" watchObservedRunningTime="2026-09-11 12:44:44.317290951 +0000 UTC m=+1067.159042133"
Sep 11 12:44:55.290565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:55.290493    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:44:55.290918 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:44:55.290767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:45:00.139456 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.139425    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"]
Sep 11 12:45:00.144176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.144154    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"]
Sep 11 12:45:00.144300 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.144268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"
Sep 11 12:45:00.147410 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.147387    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:45:00.148172 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.148152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"
Sep 11 12:45:00.151977 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.151952    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"]
Sep 11 12:45:00.156188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.156052    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"]
Sep 11 12:45:00.158649 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.158627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"
Sep 11 12:45:00.165523 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.165505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"
Sep 11 12:45:00.446241 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.446211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"]
Sep 11 12:45:00.448205 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.448177    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"]
Sep 11 12:45:00.449116 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:45:00.449084    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d11676_123d_465b_9680_b8a05df3a171.slice/crio-6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7 WatchSource:0}: Error finding container 6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7: Status 404 returned error can't find the container with id 6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7
Sep 11 12:45:00.449873 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:45:00.449853    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a3a6f2_4df5_4e04_b135_872c59d093a3.slice/crio-9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe WatchSource:0}: Error finding container 9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe: Status 404 returned error can't find the container with id 9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe
Sep 11 12:45:00.451064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:00.451048    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:45:01.368854 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:01.368814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb" event={"ID":"39a3a6f2-4df5-4e04-b135-872c59d093a3","Type":"ContainerStarted","Data":"9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe"}
Sep 11 12:45:01.370488 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:01.370443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d" event={"ID":"31d11676-123d-465b-9680-b8a05df3a171","Type":"ContainerStarted","Data":"6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7"}
Sep 11 12:45:03.382121 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:03.382082    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb" event={"ID":"39a3a6f2-4df5-4e04-b135-872c59d093a3","Type":"ContainerStarted","Data":"073ed1f5f86d6034347c54685f0cd6367896cc83f12a331b285bd40457eecbef"}
Sep 11 12:45:03.383694 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:03.383667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d" event={"ID":"31d11676-123d-465b-9680-b8a05df3a171","Type":"ContainerStarted","Data":"8e9d462ad589c47d0b8a7bb80abf61ea48dd58aefda016e70f27dfb0f7b13d73"}
Sep 11 12:45:03.397283 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:03.397241    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb" podStartSLOduration=1.285040696 podStartE2EDuration="3.397229117s" totalImagesPullingTime="2.112188421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:03.395882182 +0000 UTC m=+1086.237633318" watchObservedRunningTime="2026-09-11 12:45:03.397229117 +0000 UTC m=+1086.238980274"
Sep 11 12:45:03.409873 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:03.409837    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d" podStartSLOduration=1.300815482 podStartE2EDuration="3.409824067s" totalImagesPullingTime="2.109008585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:03.408412917 +0000 UTC m=+1086.250164065" watchObservedRunningTime="2026-09-11 12:45:03.409824067 +0000 UTC m=+1086.251575234"
Sep 11 12:45:04.390636 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:04.390490    2575 generic.go:356] "Generic (PLEG): container finished" podID="39a3a6f2-4df5-4e04-b135-872c59d093a3" containerID="073ed1f5f86d6034347c54685f0cd6367896cc83f12a331b285bd40457eecbef" exitCode=0
Sep 11 12:45:04.390636 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:04.390589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb" event={"ID":"39a3a6f2-4df5-4e04-b135-872c59d093a3","Type":"ContainerDied","Data":"073ed1f5f86d6034347c54685f0cd6367896cc83f12a331b285bd40457eecbef"}
Sep 11 12:45:04.392158 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:04.392136    2575 generic.go:356] "Generic (PLEG): container finished" podID="31d11676-123d-465b-9680-b8a05df3a171" containerID="8e9d462ad589c47d0b8a7bb80abf61ea48dd58aefda016e70f27dfb0f7b13d73" exitCode=0
Sep 11 12:45:04.392281 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:04.392201    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d" event={"ID":"31d11676-123d-465b-9680-b8a05df3a171","Type":"ContainerDied","Data":"8e9d462ad589c47d0b8a7bb80abf61ea48dd58aefda016e70f27dfb0f7b13d73"}
Sep 11 12:45:05.102941 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:05.102911    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:45:05.523690 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:05.523668    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"
Sep 11 12:45:05.552315 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:05.552289    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"
Sep 11 12:45:06.402625 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.402601    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"
Sep 11 12:45:06.402833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.402638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d" event={"ID":"31d11676-123d-465b-9680-b8a05df3a171","Type":"ContainerDied","Data":"6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7"}
Sep 11 12:45:06.402833 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.402688    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6431d78bc58f21405273570780174acf0720d9cae62e0966e613c95f9c29e1b7"
Sep 11 12:45:06.403997 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.403975    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"
Sep 11 12:45:06.404092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.403972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb" event={"ID":"39a3a6f2-4df5-4e04-b135-872c59d093a3","Type":"ContainerDied","Data":"9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe"}
Sep 11 12:45:06.404092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:06.404086    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9803b3a0d1934d0fe2cf141e8eed02a33db672bddf8c1a07429379f1559ca6fe"
Sep 11 12:45:08.931775 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:08.931743    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:45:19.882653 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:19.882617    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"]
Sep 11 12:45:19.884834 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:19.884811    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-edce0b-29818845-b942d"]
Sep 11 12:45:19.914103 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:19.914074    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:45:19.914312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:19.914288    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" podUID="4c5168fd-6559-47d3-b798-3b812e4db5ca" containerName="payload-pre-processing" containerID="cri-o://5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0" gracePeriod=30
Sep 11 12:45:20.121692 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.121655    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" podUID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" containerName="main" containerID="cri-o://024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8" gracePeriod=45
Sep 11 12:45:20.244951 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.244927    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:45:20.357210 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.357187    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:45:20.386179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.386125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkzwz\" (UniqueName: \"kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz\") pod \"4c5168fd-6559-47d3-b798-3b812e4db5ca\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") "
Sep 11 12:45:20.386179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.386177    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume\") pod \"4c5168fd-6559-47d3-b798-3b812e4db5ca\" (UID: \"4c5168fd-6559-47d3-b798-3b812e4db5ca\") "
Sep 11 12:45:20.386580 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.386555    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c5168fd-6559-47d3-b798-3b812e4db5ca" (UID: "4c5168fd-6559-47d3-b798-3b812e4db5ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:45:20.388013 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.387989    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz" (OuterVolumeSpecName: "kube-api-access-qkzwz") pod "4c5168fd-6559-47d3-b798-3b812e4db5ca" (UID: "4c5168fd-6559-47d3-b798-3b812e4db5ca"). InnerVolumeSpecName "kube-api-access-qkzwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:20.468489 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.468458    2575 generic.go:356] "Generic (PLEG): container finished" podID="4c5168fd-6559-47d3-b798-3b812e4db5ca" containerID="5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0" exitCode=0
Sep 11 12:45:20.468634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.468498    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"
Sep 11 12:45:20.468634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.468546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" event={"ID":"4c5168fd-6559-47d3-b798-3b812e4db5ca","Type":"ContainerDied","Data":"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"}
Sep 11 12:45:20.468634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.468603    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl" event={"ID":"4c5168fd-6559-47d3-b798-3b812e4db5ca","Type":"ContainerDied","Data":"4438ddff5ef5017728595982b0fc257080c89c8be0950a5b6da6ff4cbb017747"}
Sep 11 12:45:20.468634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.468631    2575 scope.go:117] "RemoveContainer" containerID="5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"
Sep 11 12:45:20.469942 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.469920    2575 generic.go:356] "Generic (PLEG): container finished" podID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" containerID="024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8" exitCode=0
Sep 11 12:45:20.470044 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.469957    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"
Sep 11 12:45:20.470044 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.469972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" event={"ID":"3ad1aa6a-3846-476f-971e-cfdd23cc91a4","Type":"ContainerDied","Data":"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"}
Sep 11 12:45:20.470044 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.470016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s" event={"ID":"3ad1aa6a-3846-476f-971e-cfdd23cc91a4","Type":"ContainerDied","Data":"1b1473ac793f07b498a2caf0a4d94b5de254d3aac8544aedf2078971d82db894"}
Sep 11 12:45:20.477078 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.477060    2575 scope.go:117] "RemoveContainer" containerID="5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"
Sep 11 12:45:20.477415 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:20.477388    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0\": container with ID starting with 5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0 not found: ID does not exist" containerID="5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"
Sep 11 12:45:20.477522 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.477425    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0"} err="failed to get container status \"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0\": rpc error: code = NotFound desc = could not find container \"5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0\": container with ID starting with 5e914915f4d9b5cdafb7042399c1b7abf9b76b6be035e786ec0794440608e3c0 not found: ID does not exist"
Sep 11 12:45:20.477522 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.477447    2575 scope.go:117] "RemoveContainer" containerID="024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"
Sep 11 12:45:20.484646 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.484632    2575 scope.go:117] "RemoveContainer" containerID="024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"
Sep 11 12:45:20.484909 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:20.484893    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8\": container with ID starting with 024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8 not found: ID does not exist" containerID="024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"
Sep 11 12:45:20.484955 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.484916    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8"} err="failed to get container status \"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8\": rpc error: code = NotFound desc = could not find container \"024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8\": container with ID starting with 024f9c8f7f85562413bee7d88f88e8f20558951e75030f09f157334c78c578f8 not found: ID does not exist"
Sep 11 12:45:20.487187 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487172    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487247 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487224    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487296 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487248    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487364 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487311    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcx4d\" (UniqueName: \"kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487364 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487354    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487520 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487398    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache\") pod \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\" (UID: \"3ad1aa6a-3846-476f-971e-cfdd23cc91a4\") "
Sep 11 12:45:20.487520 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487438    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:20.487520 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487487    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home" (OuterVolumeSpecName: "home") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:20.487639 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487599    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzwz\" (UniqueName: \"kubernetes.io/projected/4c5168fd-6559-47d3-b798-3b812e4db5ca-kube-api-access-qkzwz\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.487639 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487610    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.487639 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487618    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.487639 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487631    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5168fd-6559-47d3-b798-3b812e4db5ca-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.487760 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.487644    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:20.489363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.489319    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:45:20.489363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.489353    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm" (OuterVolumeSpecName: "dshm") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:20.489444 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.489348    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d" (OuterVolumeSpecName: "kube-api-access-kcx4d") pod "3ad1aa6a-3846-476f-971e-cfdd23cc91a4" (UID: "3ad1aa6a-3846-476f-971e-cfdd23cc91a4"). InnerVolumeSpecName "kube-api-access-kcx4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:20.493849 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.493827    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:45:20.496464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.496445    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edce0b-6c4b8f47f4-jr2kl"]
Sep 11 12:45:20.588382 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.588364    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.588382 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.588383    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.588503 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.588392    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.588503 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.588401    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcx4d\" (UniqueName: \"kubernetes.io/projected/3ad1aa6a-3846-476f-971e-cfdd23cc91a4-kube-api-access-kcx4d\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:20.791621 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.791602    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:45:20.793222 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:20.793203    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-edce0b/test-model-edce0b-kserve-549b9ff46c-c5k9s"]
Sep 11 12:45:21.629889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:21.629857    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d11676-123d-465b-9680-b8a05df3a171" path="/var/lib/kubelet/pods/31d11676-123d-465b-9680-b8a05df3a171/volumes"
Sep 11 12:45:21.630255 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:21.630228    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" path="/var/lib/kubelet/pods/3ad1aa6a-3846-476f-971e-cfdd23cc91a4/volumes"
Sep 11 12:45:21.630559 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:21.630545    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c5168fd-6559-47d3-b798-3b812e4db5ca" path="/var/lib/kubelet/pods/4c5168fd-6559-47d3-b798-3b812e4db5ca/volumes"
Sep 11 12:45:23.950210 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:23.950174    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" podUID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" containerName="main" containerID="cri-o://f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171" gracePeriod=45
Sep 11 12:45:24.193845 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.193821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:45:24.316808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316739    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.316808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.316987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316832    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.316987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316848    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjzzq\" (UniqueName: \"kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.316987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316869    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.316987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.316914    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir\") pod \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\" (UID: \"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774\") "
Sep 11 12:45:24.317228 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.317095    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache" (OuterVolumeSpecName: "model-cache") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:24.317228 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.317188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home" (OuterVolumeSpecName: "home") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:24.317228 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.317222    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.317412 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.317300    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:24.318951 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.318926    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm" (OuterVolumeSpecName: "dshm") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:45:24.319050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.318997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:45:24.319050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.319024    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq" (OuterVolumeSpecName: "kube-api-access-jjzzq") pod "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" (UID: "223fe8c7-37fa-47ea-b7f6-d6ca65a8a774"). InnerVolumeSpecName "kube-api-access-jjzzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:24.417998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.417966    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.417998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.417998    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.417998 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.418013    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzzq\" (UniqueName: \"kubernetes.io/projected/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-kube-api-access-jjzzq\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.418197 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.418023    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.418197 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.418031    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:24.491794 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.491766    2575 generic.go:356] "Generic (PLEG): container finished" podID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" containerID="f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171" exitCode=0
Sep 11 12:45:24.491893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.491810    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"
Sep 11 12:45:24.491893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.491814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" event={"ID":"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774","Type":"ContainerDied","Data":"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"}
Sep 11 12:45:24.491893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.491840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q" event={"ID":"223fe8c7-37fa-47ea-b7f6-d6ca65a8a774","Type":"ContainerDied","Data":"30664f66f3284127f7a8b5b8430e8a1129c19496a9d1108edf526f5f766a0eda"}
Sep 11 12:45:24.491893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.491855    2575 scope.go:117] "RemoveContainer" containerID="f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"
Sep 11 12:45:24.500384 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.500368    2575 scope.go:117] "RemoveContainer" containerID="f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"
Sep 11 12:45:24.500659 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:24.500640    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171\": container with ID starting with f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171 not found: ID does not exist" containerID="f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"
Sep 11 12:45:24.500710 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.500667    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171"} err="failed to get container status \"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171\": rpc error: code = NotFound desc = could not find container \"f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171\": container with ID starting with f7ba2c39cf513b757d7f8f187899ac89766f76fd3e7d69f98f38cea45ff84171 not found: ID does not exist"
Sep 11 12:45:24.512723 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.512703    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:45:24.516680 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.516660    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f06aba/test-model-f06aba-kserve-5db56d44b5-8bg5q"]
Sep 11 12:45:24.790506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.790473    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:45:24.790766 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.790738    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" podUID="a19fc866-c354-473f-88c1-7a2ca1826435" containerName="payload-processing" containerID="cri-o://4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6" gracePeriod=30
Sep 11 12:45:24.795087 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.794849    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:45:24.795087 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:24.795043    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" podUID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" containerName="payload-pre-processing" containerID="cri-o://a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c" gracePeriod=30
Sep 11 12:45:25.063412 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.063390    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:45:25.065872 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.065856    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:45:25.124372 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124344    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume\") pod \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") "
Sep 11 12:45:25.124372 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124375    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffd7n\" (UniqueName: \"kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n\") pod \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\" (UID: \"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90\") "
Sep 11 12:45:25.124570 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq94n\" (UniqueName: \"kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n\") pod \"a19fc866-c354-473f-88c1-7a2ca1826435\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") "
Sep 11 12:45:25.124570 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124459    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume\") pod \"a19fc866-c354-473f-88c1-7a2ca1826435\" (UID: \"a19fc866-c354-473f-88c1-7a2ca1826435\") "
Sep 11 12:45:25.124717 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124691    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" (UID: "daf84f24-c4d6-4abe-bf77-2be1c3cb9c90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:45:25.124995 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.124960    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a19fc866-c354-473f-88c1-7a2ca1826435" (UID: "a19fc866-c354-473f-88c1-7a2ca1826435"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:45:25.126502 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.126474    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n" (OuterVolumeSpecName: "kube-api-access-bq94n") pod "a19fc866-c354-473f-88c1-7a2ca1826435" (UID: "a19fc866-c354-473f-88c1-7a2ca1826435"). InnerVolumeSpecName "kube-api-access-bq94n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:25.126561 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.126509    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n" (OuterVolumeSpecName: "kube-api-access-ffd7n") pod "daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" (UID: "daf84f24-c4d6-4abe-bf77-2be1c3cb9c90"). InnerVolumeSpecName "kube-api-access-ffd7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:25.225124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.225100    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq94n\" (UniqueName: \"kubernetes.io/projected/a19fc866-c354-473f-88c1-7a2ca1826435-kube-api-access-bq94n\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:25.225124 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.225120    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc866-c354-473f-88c1-7a2ca1826435-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:25.225245 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.225129    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:25.225245 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.225138    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffd7n\" (UniqueName: \"kubernetes.io/projected/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90-kube-api-access-ffd7n\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:25.500253 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.500222    2575 generic.go:356] "Generic (PLEG): container finished" podID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" containerID="a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c" exitCode=0
Sep 11 12:45:25.500413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.500264    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"
Sep 11 12:45:25.500413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.500308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" event={"ID":"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90","Type":"ContainerDied","Data":"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"}
Sep 11 12:45:25.500413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.500371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2" event={"ID":"daf84f24-c4d6-4abe-bf77-2be1c3cb9c90","Type":"ContainerDied","Data":"c3140ab29c37f1e80b30a133a6a40f79ddfaf4ac6998cd425f46044e4d68e517"}
Sep 11 12:45:25.500413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.500393    2575 scope.go:117] "RemoveContainer" containerID="a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"
Sep 11 12:45:25.501610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.501589    2575 generic.go:356] "Generic (PLEG): container finished" podID="a19fc866-c354-473f-88c1-7a2ca1826435" containerID="4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6" exitCode=0
Sep 11 12:45:25.501742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.501636    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"
Sep 11 12:45:25.501742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.501636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" event={"ID":"a19fc866-c354-473f-88c1-7a2ca1826435","Type":"ContainerDied","Data":"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"}
Sep 11 12:45:25.501742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.501682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8" event={"ID":"a19fc866-c354-473f-88c1-7a2ca1826435","Type":"ContainerDied","Data":"ff2b189e64ce75111827417424f7cf0a05f01a57e5baf7cfed234d093ec16985"}
Sep 11 12:45:25.510398 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.510380    2575 scope.go:117] "RemoveContainer" containerID="a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"
Sep 11 12:45:25.510638 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:25.510621    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c\": container with ID starting with a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c not found: ID does not exist" containerID="a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"
Sep 11 12:45:25.510683 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.510647    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c"} err="failed to get container status \"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c\": rpc error: code = NotFound desc = could not find container \"a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c\": container with ID starting with a8286f214937cfcf201f525dbbb5e01485a35a3519f785337ae51cdfd8f07d7c not found: ID does not exist"
Sep 11 12:45:25.510683 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.510665    2575 scope.go:117] "RemoveContainer" containerID="4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"
Sep 11 12:45:25.525012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.518683    2575 scope.go:117] "RemoveContainer" containerID="4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"
Sep 11 12:45:25.525012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.524301    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:45:25.525165 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:25.525119    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6\": container with ID starting with 4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6 not found: ID does not exist" containerID="4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"
Sep 11 12:45:25.525386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.525213    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6"} err="failed to get container status \"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6\": rpc error: code = NotFound desc = could not find container \"4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6\": container with ID starting with 4e3b0d76d963d96a5db404aee2808f71c44ea2386f42e818d793579f592531e6 not found: ID does not exist"
Sep 11 12:45:25.528568 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.528546    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54e055b6-6dd794cdcd-c4pm2"]
Sep 11 12:45:25.538580 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.538559    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:45:25.542787 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.542761    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54e055b6-54b5bc958f-jk7b8"]
Sep 11 12:45:25.629222 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.629198    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" path="/var/lib/kubelet/pods/223fe8c7-37fa-47ea-b7f6-d6ca65a8a774/volumes"
Sep 11 12:45:25.629545 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.629532    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19fc866-c354-473f-88c1-7a2ca1826435" path="/var/lib/kubelet/pods/a19fc866-c354-473f-88c1-7a2ca1826435/volumes"
Sep 11 12:45:25.629838 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:25.629827    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" path="/var/lib/kubelet/pods/daf84f24-c4d6-4abe-bf77-2be1c3cb9c90/volumes"
Sep 11 12:45:37.472055 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.472022    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"]
Sep 11 12:45:37.486949 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.486914    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f06aba-29818845-vxrhb"]
Sep 11 12:45:37.495139 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.495107    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:45:37.495436 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.495385    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" podUID="ad600284-95bd-475f-9094-72a478483f53" containerName="payload-processing" containerID="cri-o://c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9" gracePeriod=30
Sep 11 12:45:37.631433 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.631399    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a3a6f2-4df5-4e04-b135-872c59d093a3" path="/var/lib/kubelet/pods/39a3a6f2-4df5-4e04-b135-872c59d093a3/volumes"
Sep 11 12:45:37.750654 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.750629    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:45:37.823073 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.823042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume\") pod \"ad600284-95bd-475f-9094-72a478483f53\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") "
Sep 11 12:45:37.823355 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.823079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls\") pod \"ad600284-95bd-475f-9094-72a478483f53\" (UID: \"ad600284-95bd-475f-9094-72a478483f53\") "
Sep 11 12:45:37.823440 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.823394    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad600284-95bd-475f-9094-72a478483f53" (UID: "ad600284-95bd-475f-9094-72a478483f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:45:37.825149 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.825119    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls" (OuterVolumeSpecName: "kube-api-access-vkgls") pod "ad600284-95bd-475f-9094-72a478483f53" (UID: "ad600284-95bd-475f-9094-72a478483f53"). InnerVolumeSpecName "kube-api-access-vkgls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:45:37.924457 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.924434    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad600284-95bd-475f-9094-72a478483f53-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:37.924457 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:37.924455    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/ad600284-95bd-475f-9094-72a478483f53-kube-api-access-vkgls\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:45:38.572305 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.572261    2575 generic.go:356] "Generic (PLEG): container finished" podID="ad600284-95bd-475f-9094-72a478483f53" containerID="c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9" exitCode=0
Sep 11 12:45:38.572743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.572344    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"
Sep 11 12:45:38.572743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.572354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" event={"ID":"ad600284-95bd-475f-9094-72a478483f53","Type":"ContainerDied","Data":"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"}
Sep 11 12:45:38.572743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.572416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l" event={"ID":"ad600284-95bd-475f-9094-72a478483f53","Type":"ContainerDied","Data":"288bf453f660b091d21bd341fcc832063fdf4e7c68be69daa4246ed396b12900"}
Sep 11 12:45:38.572743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.572445    2575 scope.go:117] "RemoveContainer" containerID="c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"
Sep 11 12:45:38.581690 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.581672    2575 scope.go:117] "RemoveContainer" containerID="c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"
Sep 11 12:45:38.581936 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:45:38.581922    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9\": container with ID starting with c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9 not found: ID does not exist" containerID="c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"
Sep 11 12:45:38.581976 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.581945    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9"} err="failed to get container status \"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9\": rpc error: code = NotFound desc = could not find container \"c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9\": container with ID starting with c4cd24881b4e571f3118d6a5ea592d0a20ef601fd92636176119c37cb7a1aca9 not found: ID does not exist"
Sep 11 12:45:38.593944 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.593920    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:45:38.595900 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:38.595882    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f06aba-6574688fc5-fgk6l"]
Sep 11 12:45:39.629388 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:39.629345    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad600284-95bd-475f-9094-72a478483f53" path="/var/lib/kubelet/pods/ad600284-95bd-475f-9094-72a478483f53/volumes"
Sep 11 12:45:50.039445 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039405    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:45:50.039898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039878    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d11676-123d-465b-9680-b8a05df3a171" containerName="cleanup"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039899    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d11676-123d-465b-9680-b8a05df3a171" containerName="cleanup"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039917    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19fc866-c354-473f-88c1-7a2ca1826435" containerName="payload-processing"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039926    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19fc866-c354-473f-88c1-7a2ca1826435" containerName="payload-processing"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039941    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" containerName="main"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039950    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" containerName="main"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039961    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5168fd-6559-47d3-b798-3b812e4db5ca" containerName="payload-pre-processing"
Sep 11 12:45:50.039969 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039969    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5168fd-6559-47d3-b798-3b812e4db5ca" containerName="payload-pre-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039980    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" containerName="main"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.039988    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" containerName="main"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040004    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" containerName="payload-pre-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040013    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" containerName="payload-pre-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040032    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad600284-95bd-475f-9094-72a478483f53" containerName="payload-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040041    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad600284-95bd-475f-9094-72a478483f53" containerName="payload-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040059    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a3a6f2-4df5-4e04-b135-872c59d093a3" containerName="cleanup"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040067    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a3a6f2-4df5-4e04-b135-872c59d093a3" containerName="cleanup"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040154    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a3a6f2-4df5-4e04-b135-872c59d093a3" containerName="cleanup"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040164    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d11676-123d-465b-9680-b8a05df3a171" containerName="cleanup"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040175    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="223fe8c7-37fa-47ea-b7f6-d6ca65a8a774" containerName="main"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040186    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19fc866-c354-473f-88c1-7a2ca1826435" containerName="payload-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040195    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad600284-95bd-475f-9094-72a478483f53" containerName="payload-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040203    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf84f24-c4d6-4abe-bf77-2be1c3cb9c90" containerName="payload-pre-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040213    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5168fd-6559-47d3-b798-3b812e4db5ca" containerName="payload-pre-processing"
Sep 11 12:45:50.040304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.040223    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad1aa6a-3846-476f-971e-cfdd23cc91a4" containerName="main"
Sep 11 12:45:50.044829 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.044806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.047372 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.047352    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c0bbb3-dockercfg-l8ltg\""
Sep 11 12:45:50.047493 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.047475    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c0bbb3\""
Sep 11 12:45:50.053967 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.053947    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:45:50.130148 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.130117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gfb\" (UniqueName: \"kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.130345 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.130244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.230652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.230620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gfb\" (UniqueName: \"kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.230805 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.230693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.231319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.231301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.238950 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.238925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gfb\" (UniqueName: \"kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb\") pod \"payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.355812 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.355733    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:50.626593 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:50.626559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:45:50.628073 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:45:50.628049    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee1da40_14f1_4405_b691_42dcbb7c5552.slice/crio-725b478386fc7e12237aeffcfd3da8e813444e1f1684a7ba3d23912de9ee53f8 WatchSource:0}: Error finding container 725b478386fc7e12237aeffcfd3da8e813444e1f1684a7ba3d23912de9ee53f8: Status 404 returned error can't find the container with id 725b478386fc7e12237aeffcfd3da8e813444e1f1684a7ba3d23912de9ee53f8
Sep 11 12:45:51.455218 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.455181    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:45:51.458527 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.458510    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.461714 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.461693    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ce2636-dockercfg-6n46c\""
Sep 11 12:45:51.461850 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.461745    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ce2636\""
Sep 11 12:45:51.469579 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.469556    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:45:51.543134 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.543097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjn6s\" (UniqueName: \"kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.543306 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.543147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.635478 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.635451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" event={"ID":"2ee1da40-14f1-4405-b691-42dcbb7c5552","Type":"ContainerStarted","Data":"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"}
Sep 11 12:45:51.635642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.635490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" event={"ID":"2ee1da40-14f1-4405-b691-42dcbb7c5552","Type":"ContainerStarted","Data":"725b478386fc7e12237aeffcfd3da8e813444e1f1684a7ba3d23912de9ee53f8"}
Sep 11 12:45:51.635642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.635542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:45:51.644180 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.644159    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjn6s\" (UniqueName: \"kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.644256 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.644204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.644836 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.644819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.652428 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.652398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjn6s\" (UniqueName: \"kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s\") pod \"payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:51.653246 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.653208    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" podStartSLOduration=1.653197258 podStartE2EDuration="1.653197258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:51.652385559 +0000 UTC m=+1134.494136711" watchObservedRunningTime="2026-09-11 12:45:51.653197258 +0000 UTC m=+1134.494948413"
Sep 11 12:45:51.769606 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:51.769528    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:52.043172 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:52.043100    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:45:52.046293 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:45:52.046265    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d85ba6f_e2dd_46ea_adc0_a326a18c6377.slice/crio-1fec2f38337c05c0057cbf073f29308557bcbe7ba29f70192b5746fe8dfd3854 WatchSource:0}: Error finding container 1fec2f38337c05c0057cbf073f29308557bcbe7ba29f70192b5746fe8dfd3854: Status 404 returned error can't find the container with id 1fec2f38337c05c0057cbf073f29308557bcbe7ba29f70192b5746fe8dfd3854
Sep 11 12:45:52.641208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:52.641176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" event={"ID":"7d85ba6f-e2dd-46ea-adc0-a326a18c6377","Type":"ContainerStarted","Data":"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"}
Sep 11 12:45:52.641642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:52.641226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" event={"ID":"7d85ba6f-e2dd-46ea-adc0-a326a18c6377","Type":"ContainerStarted","Data":"1fec2f38337c05c0057cbf073f29308557bcbe7ba29f70192b5746fe8dfd3854"}
Sep 11 12:45:52.641642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:52.641444    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:45:52.664846 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:45:52.664800    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" podStartSLOduration=1.664787972 podStartE2EDuration="1.664787972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:52.663618181 +0000 UTC m=+1135.505369324" watchObservedRunningTime="2026-09-11 12:45:52.664787972 +0000 UTC m=+1135.506539076"
Sep 11 12:46:02.643375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:02.643312    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:46:03.648142 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:03.648107    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:46:18.196506 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.196475    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:46:18.201272 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.201252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.203873 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.203852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1337fa-dockercfg-4z957\""
Sep 11 12:46:18.203991 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.203888    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1337fa\""
Sep 11 12:46:18.210856 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.210836    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:46:18.228577 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.228555    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:46:18.231751 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.231736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.243207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.243185    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:46:18.366372 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.366314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.366528 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.366444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgdk\" (UniqueName: \"kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.366528 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.366497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.366603 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.366526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgdz\" (UniqueName: \"kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.467085 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.467085 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgdk\" (UniqueName: \"kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.467257 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.467257 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgdz\" (UniqueName: \"kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.467960 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.468031 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.467935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.477205 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.477174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgdk\" (UniqueName: \"kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk\") pod \"payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.477303 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.477238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgdz\" (UniqueName: \"kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz\") pod \"payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.512308 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.512281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:18.540586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.540552    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:18.789376 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.789346    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:46:18.791734 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:46:18.791713    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3448dcf6_d924_4a92_8861_614c72d6b31e.slice/crio-ae44d55826a7c4d2cc6c0c2f0557e55bc5ea24396b724fcaf2f299a11f855c1d WatchSource:0}: Error finding container ae44d55826a7c4d2cc6c0c2f0557e55bc5ea24396b724fcaf2f299a11f855c1d: Status 404 returned error can't find the container with id ae44d55826a7c4d2cc6c0c2f0557e55bc5ea24396b724fcaf2f299a11f855c1d
Sep 11 12:46:18.828747 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:18.828727    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:46:18.831179 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:46:18.831151    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433100f4_fe18_4234_8e37_906dc914cd95.slice/crio-14a3068881f6dc33a9f9ee810aa6fad8c141e380a0877ff4b53d58c58235bfb5 WatchSource:0}: Error finding container 14a3068881f6dc33a9f9ee810aa6fad8c141e380a0877ff4b53d58c58235bfb5: Status 404 returned error can't find the container with id 14a3068881f6dc33a9f9ee810aa6fad8c141e380a0877ff4b53d58c58235bfb5
Sep 11 12:46:19.598607 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.598572    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:46:19.602088 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.602067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.604798 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.604773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5db8fe-dockercfg-nfkh2\""
Sep 11 12:46:19.604798 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.604791    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5db8fe\""
Sep 11 12:46:19.613509 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.613483    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:46:19.631790 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.631766    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:46:19.634962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.634945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.644255 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.644231    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:46:19.767313 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.767275    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" event={"ID":"3448dcf6-d924-4a92-8861-614c72d6b31e","Type":"ContainerStarted","Data":"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"}
Sep 11 12:46:19.767504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.767394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:19.767504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.767426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" event={"ID":"3448dcf6-d924-4a92-8861-614c72d6b31e","Type":"ContainerStarted","Data":"ae44d55826a7c4d2cc6c0c2f0557e55bc5ea24396b724fcaf2f299a11f855c1d"}
Sep 11 12:46:19.768958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.768930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" event={"ID":"433100f4-fe18-4234-8e37-906dc914cd95","Type":"ContainerStarted","Data":"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"}
Sep 11 12:46:19.768958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.768970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" event={"ID":"433100f4-fe18-4234-8e37-906dc914cd95","Type":"ContainerStarted","Data":"14a3068881f6dc33a9f9ee810aa6fad8c141e380a0877ff4b53d58c58235bfb5"}
Sep 11 12:46:19.769164 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.769054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:19.777233 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.777208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.777317 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.777248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflff\" (UniqueName: \"kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.777387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.777344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9lg\" (UniqueName: \"kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.777387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.777380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.785165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.785126    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" podStartSLOduration=1.785111825 podStartE2EDuration="1.785111825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:46:19.78317406 +0000 UTC m=+1162.624925206" watchObservedRunningTime="2026-09-11 12:46:19.785111825 +0000 UTC m=+1162.626862970"
Sep 11 12:46:19.799843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.799797    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" podStartSLOduration=1.799782497 podStartE2EDuration="1.799782497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:46:19.798269139 +0000 UTC m=+1162.640020287" watchObservedRunningTime="2026-09-11 12:46:19.799782497 +0000 UTC m=+1162.641533684"
Sep 11 12:46:19.878168 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.878380 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.878380 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflff\" (UniqueName: \"kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.878501 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9lg\" (UniqueName: \"kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.878987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.878987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.878975    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.886251 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.886225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflff\" (UniqueName: \"kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff\") pod \"payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:19.886522 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.886506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9lg\" (UniqueName: \"kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg\") pod \"payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.914291 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.914269    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:19.945439 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:19.945412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:20.188635 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.188610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:46:20.191490 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:46:20.191461    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46a80e2_92b1_47e4_84a5_df038f274078.slice/crio-45c6f4759312233d4701f2e0bc9074718d4626369008a70727afee324c52de78 WatchSource:0}: Error finding container 45c6f4759312233d4701f2e0bc9074718d4626369008a70727afee324c52de78: Status 404 returned error can't find the container with id 45c6f4759312233d4701f2e0bc9074718d4626369008a70727afee324c52de78
Sep 11 12:46:20.224852 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.224830    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:46:20.226733 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:46:20.226711    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744aefc9_2242_4515_8e48_a1de92f24c02.slice/crio-d7b1b58d9033336e7edac2074a326dfb44695bd695ffc6a8ad5f03309d3c8898 WatchSource:0}: Error finding container d7b1b58d9033336e7edac2074a326dfb44695bd695ffc6a8ad5f03309d3c8898: Status 404 returned error can't find the container with id d7b1b58d9033336e7edac2074a326dfb44695bd695ffc6a8ad5f03309d3c8898
Sep 11 12:46:20.776203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.776163    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" event={"ID":"744aefc9-2242-4515-8e48-a1de92f24c02","Type":"ContainerStarted","Data":"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"}
Sep 11 12:46:20.776716 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.776235    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:20.776716 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.776259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" event={"ID":"744aefc9-2242-4515-8e48-a1de92f24c02","Type":"ContainerStarted","Data":"d7b1b58d9033336e7edac2074a326dfb44695bd695ffc6a8ad5f03309d3c8898"}
Sep 11 12:46:20.777889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.777852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" event={"ID":"a46a80e2-92b1-47e4-84a5-df038f274078","Type":"ContainerStarted","Data":"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"}
Sep 11 12:46:20.778042 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.777904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" event={"ID":"a46a80e2-92b1-47e4-84a5-df038f274078","Type":"ContainerStarted","Data":"45c6f4759312233d4701f2e0bc9074718d4626369008a70727afee324c52de78"}
Sep 11 12:46:20.778042 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.777946    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:20.797118 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.797065    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" podStartSLOduration=1.797050198 podStartE2EDuration="1.797050198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:46:20.795580597 +0000 UTC m=+1163.637331740" watchObservedRunningTime="2026-09-11 12:46:20.797050198 +0000 UTC m=+1163.638801350"
Sep 11 12:46:20.813634 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:20.813585    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" podStartSLOduration=1.81356822 podStartE2EDuration="1.81356822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:46:20.811435872 +0000 UTC m=+1163.653187021" watchObservedRunningTime="2026-09-11 12:46:20.81356822 +0000 UTC m=+1163.655319345"
Sep 11 12:46:30.780167 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:30.780136    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:46:30.780578 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:30.780206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:46:31.785164 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:31.785130    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:46:31.785635 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:31.785305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:46:52.102437 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.102400    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:46:52.107375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.107354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.110099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.110077    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"openshift-service-ca.crt\""
Sep 11 12:46:52.111426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.111409    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"default-dockercfg-jlqlz\""
Sep 11 12:46:52.111526 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.111411    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"auth-test-model-c0bbb3-kserve-self-signed-certs\""
Sep 11 12:46:52.111526 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.111414    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"kube-root-ca.crt\""
Sep 11 12:46:52.115010 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.114989    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:46:52.149319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149290    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.149468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.149468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wcp\" (UniqueName: \"kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.149468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.149468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.149630 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.149482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wcp\" (UniqueName: \"kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250352 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250587    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250700 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250629    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.250700 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.250657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.252338 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.252309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.252616 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.252599    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.269545 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.269518    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wcp\" (UniqueName: \"kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp\") pod \"auth-test-model-c0bbb3-kserve-69cc67b679-x429l\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.418858 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.418770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:46:52.690091 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.690067    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:46:52.692569 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:46:52.692543    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadaee487_10ef_4f83_9a30_4a73b021d6cb.slice/crio-073abedf2640efa27aef3cae33cd1cf55f08b2da1c97bafe36fa2b4548fe8b5b WatchSource:0}: Error finding container 073abedf2640efa27aef3cae33cd1cf55f08b2da1c97bafe36fa2b4548fe8b5b: Status 404 returned error can't find the container with id 073abedf2640efa27aef3cae33cd1cf55f08b2da1c97bafe36fa2b4548fe8b5b
Sep 11 12:46:52.932726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.932684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" event={"ID":"adaee487-10ef-4f83-9a30-4a73b021d6cb","Type":"ContainerStarted","Data":"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"}
Sep 11 12:46:52.932967 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.932737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" event={"ID":"adaee487-10ef-4f83-9a30-4a73b021d6cb","Type":"ContainerStarted","Data":"073abedf2640efa27aef3cae33cd1cf55f08b2da1c97bafe36fa2b4548fe8b5b"}
Sep 11 12:46:52.951321 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:46:52.951226    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" podStartSLOduration=0.951210215 podStartE2EDuration="951.210215ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:46:52.949105369 +0000 UTC m=+1195.790856516" watchObservedRunningTime="2026-09-11 12:46:52.951210215 +0000 UTC m=+1195.792961340"
Sep 11 12:47:02.419234 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:02.419196    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:47:02.419234 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:02.419243    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:47:02.431501 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:02.431473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:47:02.995104 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:02.995071    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:47:21.904777 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.904743    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:47:21.907230 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.907214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.911499 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.911475    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"openshift-service-ca.crt\""
Sep 11 12:47:21.911622 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.911479    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"default-dockercfg-672t6\""
Sep 11 12:47:21.911622 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.911596    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"auth-test-model-1337fa-kserve-self-signed-certs\""
Sep 11 12:47:21.911739 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.911653    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"kube-root-ca.crt\""
Sep 11 12:47:21.918303 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.918278    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:47:21.994309 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9xs\" (UniqueName: \"kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.994479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994319    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.994479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.994479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.994479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:21.994479 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:21.994462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095693 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9xs\" (UniqueName: \"kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.095887 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.095786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.096131 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.096107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.096170 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.096147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.096203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.096177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.097881 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.097854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.098213 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.098195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.103926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.103903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9xs\" (UniqueName: \"kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs\") pod \"auth-test-model-1337fa-kserve-77fc57dbcf-mchgb\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.216742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.216716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:22.493456 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:22.493432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:47:22.495361 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:47:22.495317    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa863155_47af_405a_8cc2_acc41ce388d8.slice/crio-662b0eaa9b51a2bd08882e2d1160c59eda2662a67d6821ac3c9b5069a1975846 WatchSource:0}: Error finding container 662b0eaa9b51a2bd08882e2d1160c59eda2662a67d6821ac3c9b5069a1975846: Status 404 returned error can't find the container with id 662b0eaa9b51a2bd08882e2d1160c59eda2662a67d6821ac3c9b5069a1975846
Sep 11 12:47:23.081044 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:23.081002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" event={"ID":"fa863155-47af-405a-8cc2-acc41ce388d8","Type":"ContainerStarted","Data":"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"}
Sep 11 12:47:23.081449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:23.081053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" event={"ID":"fa863155-47af-405a-8cc2-acc41ce388d8","Type":"ContainerStarted","Data":"662b0eaa9b51a2bd08882e2d1160c59eda2662a67d6821ac3c9b5069a1975846"}
Sep 11 12:47:23.102846 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:23.102793    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" podStartSLOduration=2.10277142 podStartE2EDuration="2.10277142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:47:23.101638669 +0000 UTC m=+1225.943389810" watchObservedRunningTime="2026-09-11 12:47:23.10277142 +0000 UTC m=+1225.944522545"
Sep 11 12:47:32.217187 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:32.217152    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:32.217668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:32.217202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:32.229284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:32.229266    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:33.138900 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:33.138872    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:47:41.386104 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.386070    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:47:41.389485 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.389465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.392681 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.392660    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3fae345\"/\"openshift-service-ca.crt\""
Sep 11 12:47:41.392883 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.392869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3fae345\"/\"default-dockercfg-r5b7w\""
Sep 11 12:47:41.394176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.394158    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3fae345\"/\"e2e-lifecycle-model-f3fae345-kserve-self-signed-certs\""
Sep 11 12:47:41.394236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.394158    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3fae345\"/\"kube-root-ca.crt\""
Sep 11 12:47:41.403050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.403024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:47:41.464264 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.464477 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464295    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.464477 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.464477 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.464477 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.464694 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.464494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565484 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565856 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565912 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.565986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.565970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.566075 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.566058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.567802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.567782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.568090 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.568071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.573877 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.573855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2\") pod \"e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") " pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.700371 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.700312    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:41.977054 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:41.977023    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:47:41.979421 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:47:41.979393    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf0838ac_8a11_4a11_82f8_65eca9fa6505.slice/crio-ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e WatchSource:0}: Error finding container ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e: Status 404 returned error can't find the container with id ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e
Sep 11 12:47:42.171391 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:42.171347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" event={"ID":"af0838ac-8a11-4a11-82f8-65eca9fa6505","Type":"ContainerStarted","Data":"6792b2e4bf8ef4c2344b8e83e42c9e27b32fafaf992359a5d98f90e204405cd4"}
Sep 11 12:47:42.171586 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:42.171400    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" event={"ID":"af0838ac-8a11-4a11-82f8-65eca9fa6505","Type":"ContainerStarted","Data":"ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e"}
Sep 11 12:47:42.193503 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:42.193453    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" podStartSLOduration=1.193437884 podStartE2EDuration="1.193437884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:47:42.192507256 +0000 UTC m=+1245.034258396" watchObservedRunningTime="2026-09-11 12:47:42.193437884 +0000 UTC m=+1245.035189009"
Sep 11 12:47:51.700958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:51.700866    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:51.700958 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:51.700925    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:51.713290 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:51.713267    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:47:52.231844 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:47:52.231815    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:48:25.564813 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.564781    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"]
Sep 11 12:48:25.568652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.568634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:25.571529 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.571505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:48:25.571645 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.571505    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:48:25.576312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.576289    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"]
Sep 11 12:48:25.667110 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.667078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x\" (UID: \"4262e6d9-69e2-482c-a30b-7bdadf0535c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:25.767516 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.767478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x\" (UID: \"4262e6d9-69e2-482c-a30b-7bdadf0535c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:25.768084 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.768067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x\" (UID: \"4262e6d9-69e2-482c-a30b-7bdadf0535c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:25.880846 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:25.880765    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:26.151884 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:26.151860    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"]
Sep 11 12:48:26.161740 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:48:26.153976    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4262e6d9_69e2_482c_a30b_7bdadf0535c1.slice/crio-9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583 WatchSource:0}: Error finding container 9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583: Status 404 returned error can't find the container with id 9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583
Sep 11 12:48:26.387451 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:26.387419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x" event={"ID":"4262e6d9-69e2-482c-a30b-7bdadf0535c1","Type":"ContainerStarted","Data":"9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583"}
Sep 11 12:48:27.393256 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:27.393219    2575 generic.go:356] "Generic (PLEG): container finished" podID="4262e6d9-69e2-482c-a30b-7bdadf0535c1" containerID="b30514e9bc2b9fc9ae41fb8a1645384b4eb2c2874bc7814e6bff86ab79f05b51" exitCode=0
Sep 11 12:48:27.393681 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:27.393311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x" event={"ID":"4262e6d9-69e2-482c-a30b-7bdadf0535c1","Type":"ContainerDied","Data":"b30514e9bc2b9fc9ae41fb8a1645384b4eb2c2874bc7814e6bff86ab79f05b51"}
Sep 11 12:48:28.525592 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:28.525569    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:28.591207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:28.591183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca\") pod \"4262e6d9-69e2-482c-a30b-7bdadf0535c1\" (UID: \"4262e6d9-69e2-482c-a30b-7bdadf0535c1\") "
Sep 11 12:48:28.591524 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:28.591502    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4262e6d9-69e2-482c-a30b-7bdadf0535c1" (UID: "4262e6d9-69e2-482c-a30b-7bdadf0535c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:28.692312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:28.692287    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4262e6d9-69e2-482c-a30b-7bdadf0535c1-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:48:29.404035 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:29.404001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x" event={"ID":"4262e6d9-69e2-482c-a30b-7bdadf0535c1","Type":"ContainerDied","Data":"9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583"}
Sep 11 12:48:29.404204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:29.404047    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c5119a74ea9f6634a01c52a2f9972a0345974524d9e388c3dbf3c08a79b2583"
Sep 11 12:48:29.404204 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:29.404016    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"
Sep 11 12:48:41.309062 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:41.309025    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"]
Sep 11 12:48:41.311910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:41.311887    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3fae345-s7f5x"]
Sep 11 12:48:41.630092 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:41.630015    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4262e6d9-69e2-482c-a30b-7bdadf0535c1" path="/var/lib/kubelet/pods/4262e6d9-69e2-482c-a30b-7bdadf0535c1/volumes"
Sep 11 12:48:49.323393 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:48:49.323354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:49:04.342627 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.342551    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" podUID="af0838ac-8a11-4a11-82f8-65eca9fa6505" containerName="main" containerID="cri-o://6792b2e4bf8ef4c2344b8e83e42c9e27b32fafaf992359a5d98f90e204405cd4" gracePeriod=45
Sep 11 12:49:04.593794 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.593723    2575 generic.go:356] "Generic (PLEG): container finished" podID="af0838ac-8a11-4a11-82f8-65eca9fa6505" containerID="6792b2e4bf8ef4c2344b8e83e42c9e27b32fafaf992359a5d98f90e204405cd4" exitCode=0
Sep 11 12:49:04.593932 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.593788    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" event={"ID":"af0838ac-8a11-4a11-82f8-65eca9fa6505","Type":"ContainerDied","Data":"6792b2e4bf8ef4c2344b8e83e42c9e27b32fafaf992359a5d98f90e204405cd4"}
Sep 11 12:49:04.593932 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.593833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm" event={"ID":"af0838ac-8a11-4a11-82f8-65eca9fa6505","Type":"ContainerDied","Data":"ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e"}
Sep 11 12:49:04.593932 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.593845    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9ca0ebb448bc78086d440b1825073521280230d559d48a6f81bca70ca40d3e"
Sep 11 12:49:04.597161 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.597145    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:49:04.710512 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710479    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710545    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710666 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710624    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710788 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710676    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710788 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710725    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2\") pod \"af0838ac-8a11-4a11-82f8-65eca9fa6505\" (UID: \"af0838ac-8a11-4a11-82f8-65eca9fa6505\") "
Sep 11 12:49:04.710884 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710764    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home" (OuterVolumeSpecName: "home") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:04.710884 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710844    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache" (OuterVolumeSpecName: "model-cache") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:04.711000 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.710903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:04.711096 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.711070    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:04.711145 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.711099    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:04.711145 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.711112    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:04.712758 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.712731    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2" (OuterVolumeSpecName: "kube-api-access-72bg2") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "kube-api-access-72bg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:49:04.712843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.712780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:49:04.712843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.712814    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm" (OuterVolumeSpecName: "dshm") pod "af0838ac-8a11-4a11-82f8-65eca9fa6505" (UID: "af0838ac-8a11-4a11-82f8-65eca9fa6505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:04.812070 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.812049    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0838ac-8a11-4a11-82f8-65eca9fa6505-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:04.812070 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.812069    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0838ac-8a11-4a11-82f8-65eca9fa6505-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:04.812193 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:04.812080    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/af0838ac-8a11-4a11-82f8-65eca9fa6505-kube-api-access-72bg2\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:05.598692 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:05.598655    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"
Sep 11 12:49:05.619622 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:05.619594    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:49:05.629251 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:05.629230    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f3fae345/e2e-lifecycle-model-f3fae345-kserve-797bb5dc4c-7z9sm"]
Sep 11 12:49:07.630077 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:07.630038    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0838ac-8a11-4a11-82f8-65eca9fa6505" path="/var/lib/kubelet/pods/af0838ac-8a11-4a11-82f8-65eca9fa6505/volumes"
Sep 11 12:49:35.151827 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:35.151728    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:49:36.389884 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:36.389855    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:49:50.170003 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.169934    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" podUID="adaee487-10ef-4f83-9a30-4a73b021d6cb" containerName="main" containerID="cri-o://199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715" gracePeriod=45
Sep 11 12:49:50.408061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.408038    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:49:50.481119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481091    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481250 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481162    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4wcp\" (UniqueName: \"kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481250 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481376 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481302    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481438 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481438 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481419    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir\") pod \"adaee487-10ef-4f83-9a30-4a73b021d6cb\" (UID: \"adaee487-10ef-4f83-9a30-4a73b021d6cb\") "
Sep 11 12:49:50.481576 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481549    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:50.481635 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481566    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home" (OuterVolumeSpecName: "home") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:50.481744 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481712    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:50.481848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481810    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.481848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481828    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.481848 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.481841    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.483381 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.483355    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp" (OuterVolumeSpecName: "kube-api-access-s4wcp") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "kube-api-access-s4wcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:49:50.483468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.483376    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:49:50.483468 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.483390    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm" (OuterVolumeSpecName: "dshm") pod "adaee487-10ef-4f83-9a30-4a73b021d6cb" (UID: "adaee487-10ef-4f83-9a30-4a73b021d6cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:50.582997 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.582978    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaee487-10ef-4f83-9a30-4a73b021d6cb-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.582997 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.582998    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaee487-10ef-4f83-9a30-4a73b021d6cb-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.583134 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.583007    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4wcp\" (UniqueName: \"kubernetes.io/projected/adaee487-10ef-4f83-9a30-4a73b021d6cb-kube-api-access-s4wcp\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:50.818130 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.818052    2575 generic.go:356] "Generic (PLEG): container finished" podID="adaee487-10ef-4f83-9a30-4a73b021d6cb" containerID="199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715" exitCode=0
Sep 11 12:49:50.818130 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.818105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" event={"ID":"adaee487-10ef-4f83-9a30-4a73b021d6cb","Type":"ContainerDied","Data":"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"}
Sep 11 12:49:50.818289 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.818139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l" event={"ID":"adaee487-10ef-4f83-9a30-4a73b021d6cb","Type":"ContainerDied","Data":"073abedf2640efa27aef3cae33cd1cf55f08b2da1c97bafe36fa2b4548fe8b5b"}
Sep 11 12:49:50.818289 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.818155    2575 scope.go:117] "RemoveContainer" containerID="199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"
Sep 11 12:49:50.818289 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.818108    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"
Sep 11 12:49:50.827403 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.827385    2575 scope.go:117] "RemoveContainer" containerID="199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"
Sep 11 12:49:50.827673 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:49:50.827652    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715\": container with ID starting with 199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715 not found: ID does not exist" containerID="199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"
Sep 11 12:49:50.827738 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.827681    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715"} err="failed to get container status \"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715\": rpc error: code = NotFound desc = could not find container \"199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715\": container with ID starting with 199339b4e51d53c8ccc55f966bc61459b69782af24cedd25dc68bb453dabe715 not found: ID does not exist"
Sep 11 12:49:50.839544 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.839523    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:49:50.843746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:50.843726    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/auth-test-model-c0bbb3-kserve-69cc67b679-x429l"]
Sep 11 12:49:51.408381 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.408345    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" podUID="fa863155-47af-405a-8cc2-acc41ce388d8" containerName="main" containerID="cri-o://f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48" gracePeriod=45
Sep 11 12:49:51.631937 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.631910    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adaee487-10ef-4f83-9a30-4a73b021d6cb" path="/var/lib/kubelet/pods/adaee487-10ef-4f83-9a30-4a73b021d6cb/volumes"
Sep 11 12:49:51.658682 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.658634    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:49:51.800021 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.799989    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800032    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800088    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800142    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg9xs\" (UniqueName: \"kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800182 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs\") pod \"fa863155-47af-405a-8cc2-acc41ce388d8\" (UID: \"fa863155-47af-405a-8cc2-acc41ce388d8\") "
Sep 11 12:49:51.800413 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800279    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home" (OuterVolumeSpecName: "home") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:51.800526 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800490    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:51.800526 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800469    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:51.800526 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.800506    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:51.802272 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.802252    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:49:51.802391 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.802340    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm" (OuterVolumeSpecName: "dshm") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:49:51.802391 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.802345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs" (OuterVolumeSpecName: "kube-api-access-gg9xs") pod "fa863155-47af-405a-8cc2-acc41ce388d8" (UID: "fa863155-47af-405a-8cc2-acc41ce388d8"). InnerVolumeSpecName "kube-api-access-gg9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:49:51.824428 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.824400    2575 generic.go:356] "Generic (PLEG): container finished" podID="fa863155-47af-405a-8cc2-acc41ce388d8" containerID="f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48" exitCode=0
Sep 11 12:49:51.824531 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.824436    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"
Sep 11 12:49:51.824531 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.824482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" event={"ID":"fa863155-47af-405a-8cc2-acc41ce388d8","Type":"ContainerDied","Data":"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"}
Sep 11 12:49:51.824531 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.824522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb" event={"ID":"fa863155-47af-405a-8cc2-acc41ce388d8","Type":"ContainerDied","Data":"662b0eaa9b51a2bd08882e2d1160c59eda2662a67d6821ac3c9b5069a1975846"}
Sep 11 12:49:51.824690 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.824544    2575 scope.go:117] "RemoveContainer" containerID="f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"
Sep 11 12:49:51.833300 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.833283    2575 scope.go:117] "RemoveContainer" containerID="f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"
Sep 11 12:49:51.833608 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:49:51.833591    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48\": container with ID starting with f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48 not found: ID does not exist" containerID="f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"
Sep 11 12:49:51.833672 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.833613    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48"} err="failed to get container status \"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48\": rpc error: code = NotFound desc = could not find container \"f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48\": container with ID starting with f23447cf837574d60abc7a75eb6b80296fc24ba3aefb1c1c74b26b547e510c48 not found: ID does not exist"
Sep 11 12:49:51.845201 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.845180    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:49:51.848920 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.848901    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/auth-test-model-1337fa-kserve-77fc57dbcf-mchgb"]
Sep 11 12:49:51.901572 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.901547    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:51.901572 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.901565    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:51.901691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.901574    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9xs\" (UniqueName: \"kubernetes.io/projected/fa863155-47af-405a-8cc2-acc41ce388d8-kube-api-access-gg9xs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:51.901691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.901584    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa863155-47af-405a-8cc2-acc41ce388d8-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:51.901691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:51.901593    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa863155-47af-405a-8cc2-acc41ce388d8-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:49:53.629926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:49:53.629892    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa863155-47af-405a-8cc2-acc41ce388d8" path="/var/lib/kubelet/pods/fa863155-47af-405a-8cc2-acc41ce388d8/volumes"
Sep 11 12:50:19.099802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.099767    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100118    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa863155-47af-405a-8cc2-acc41ce388d8" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100128    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa863155-47af-405a-8cc2-acc41ce388d8" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100140    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaee487-10ef-4f83-9a30-4a73b021d6cb" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100146    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaee487-10ef-4f83-9a30-4a73b021d6cb" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100155    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4262e6d9-69e2-482c-a30b-7bdadf0535c1" containerName="revoke-keys"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100160    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4262e6d9-69e2-482c-a30b-7bdadf0535c1" containerName="revoke-keys"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100175    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0838ac-8a11-4a11-82f8-65eca9fa6505" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100180    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0838ac-8a11-4a11-82f8-65eca9fa6505" containerName="main"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100238    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4262e6d9-69e2-482c-a30b-7bdadf0535c1" containerName="revoke-keys"
Sep 11 12:50:19.100243 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100246    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa863155-47af-405a-8cc2-acc41ce388d8" containerName="main"
Sep 11 12:50:19.100583 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100253    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0838ac-8a11-4a11-82f8-65eca9fa6505" containerName="main"
Sep 11 12:50:19.100583 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.100259    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaee487-10ef-4f83-9a30-4a73b021d6cb" containerName="main"
Sep 11 12:50:19.104595 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.104577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.108419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.108394    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"openshift-service-ca.crt\""
Sep 11 12:50:19.108419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.108412    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"sub-test-model-1337fa-kserve-self-signed-certs\""
Sep 11 12:50:19.108609 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.108413    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"default-dockercfg-672t6\""
Sep 11 12:50:19.108609 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.108502    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"kube-root-ca.crt\""
Sep 11 12:50:19.113551 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.113528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:50:19.115556 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115533    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.115662 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.115662 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.115662 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rrs\" (UniqueName: \"kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.115802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115693    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.115802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.115784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217018 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.216989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217018 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217219 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217219 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217219 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rrs\" (UniqueName: \"kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217219 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217463 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217463 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.217544 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.217532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.219115 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.219094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.219417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.219399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.226420 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.226393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rrs\" (UniqueName: \"kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs\") pod \"sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.414596 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.414540    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:19.684442 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.684371    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:50:19.687809 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:50:19.687775    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c70d873_7491_4380_8dc5_ff5d89a6cba8.slice/crio-8b5c5a9f71ae56c1051a838647b2494ddcf67cd4b07dcf9c0f14d28c9610aee1 WatchSource:0}: Error finding container 8b5c5a9f71ae56c1051a838647b2494ddcf67cd4b07dcf9c0f14d28c9610aee1: Status 404 returned error can't find the container with id 8b5c5a9f71ae56c1051a838647b2494ddcf67cd4b07dcf9c0f14d28c9610aee1
Sep 11 12:50:19.960856 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.960825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" event={"ID":"3c70d873-7491-4380-8dc5-ff5d89a6cba8","Type":"ContainerStarted","Data":"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"}
Sep 11 12:50:19.961015 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:19.960868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" event={"ID":"3c70d873-7491-4380-8dc5-ff5d89a6cba8","Type":"ContainerStarted","Data":"8b5c5a9f71ae56c1051a838647b2494ddcf67cd4b07dcf9c0f14d28c9610aee1"}
Sep 11 12:50:29.415036 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:29.415004    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:29.415577 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:29.415058    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:29.427400 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:29.427376    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:50:29.450546 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:29.450506    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" podStartSLOduration=10.450494163 podStartE2EDuration="10.450494163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:50:19.983749771 +0000 UTC m=+1402.825500909" watchObservedRunningTime="2026-09-11 12:50:29.450494163 +0000 UTC m=+1412.292245288"
Sep 11 12:50:30.020754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:50:30.020722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:51:07.311971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.311888    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:51:07.315149 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.315134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.317918 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.317897    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1a4a12\""
Sep 11 12:51:07.329426 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.329405    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:51:07.349215 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.349189    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:51:07.352300 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.352285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.355318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.355295    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1a4a12-dockercfg-ht9hj\""
Sep 11 12:51:07.373297 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.373276    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:51:07.433839 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.433796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.434017 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.433900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcwww\" (UniqueName: \"kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.534750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.534711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.534923 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.534756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.534923 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.534810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcwww\" (UniqueName: \"kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.534923 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.534833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcht\" (UniqueName: \"kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.535504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.535484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.543799 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.543778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcwww\" (UniqueName: \"kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww\") pod \"payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.624517 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.624450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:07.636153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.636125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.636287 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.636193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcht\" (UniqueName: \"kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.637012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.636989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.644650 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.644627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcht\" (UniqueName: \"kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht\") pod \"payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") " pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.661237 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.661214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:07.898425 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.898396    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:51:07.900347 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:51:07.900301    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1ce9e3_bcbb_42a1_9758_e0a30421ca99.slice/crio-cc4c35075d0d1c8008b4ba33ca7f460103cdff729e9c47cb4a7c85e6463e0335 WatchSource:0}: Error finding container cc4c35075d0d1c8008b4ba33ca7f460103cdff729e9c47cb4a7c85e6463e0335: Status 404 returned error can't find the container with id cc4c35075d0d1c8008b4ba33ca7f460103cdff729e9c47cb4a7c85e6463e0335
Sep 11 12:51:07.941271 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:07.941242    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:51:07.943152 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:51:07.943123    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae097f9c_5bb1_4b77_a3c2_b18ac5fe2277.slice/crio-5db02737b3635e8d67bd330f00869fcd1dd592c7499674d5ef8818edfaa035c9 WatchSource:0}: Error finding container 5db02737b3635e8d67bd330f00869fcd1dd592c7499674d5ef8818edfaa035c9: Status 404 returned error can't find the container with id 5db02737b3635e8d67bd330f00869fcd1dd592c7499674d5ef8818edfaa035c9
Sep 11 12:51:08.202823 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.202786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" event={"ID":"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99","Type":"ContainerStarted","Data":"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"}
Sep 11 12:51:08.203005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.202852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" event={"ID":"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99","Type":"ContainerStarted","Data":"cc4c35075d0d1c8008b4ba33ca7f460103cdff729e9c47cb4a7c85e6463e0335"}
Sep 11 12:51:08.203005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.202897    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:08.204289 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.204264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" event={"ID":"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277","Type":"ContainerStarted","Data":"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"}
Sep 11 12:51:08.204418 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.204315    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:08.204418 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.204359    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" event={"ID":"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277","Type":"ContainerStarted","Data":"5db02737b3635e8d67bd330f00869fcd1dd592c7499674d5ef8818edfaa035c9"}
Sep 11 12:51:08.223721 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.223678    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" podStartSLOduration=1.223664994 podStartE2EDuration="1.223664994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:51:08.223211827 +0000 UTC m=+1451.064962974" watchObservedRunningTime="2026-09-11 12:51:08.223664994 +0000 UTC m=+1451.065416120"
Sep 11 12:51:08.242172 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:08.242126    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" podStartSLOduration=1.242110299 podStartE2EDuration="1.242110299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:51:08.240593059 +0000 UTC m=+1451.082344198" watchObservedRunningTime="2026-09-11 12:51:08.242110299 +0000 UTC m=+1451.083861425"
Sep 11 12:51:19.211283 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:19.211250    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:51:19.211838 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:19.211315    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:51:23.611020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:23.610986    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:51:30.686475 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.686441    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:51:30.689757 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.689739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.711031 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.711010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"openshift-service-ca.crt\""
Sep 11 12:51:30.712196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.712181    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"kube-root-ca.crt\""
Sep 11 12:51:30.712267 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.712188    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"rate-test-model-c0bbb3-kserve-self-signed-certs\""
Sep 11 12:51:30.714558 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:51:30.714531    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-jlqlz\" is forbidden: User \"system:node:ip-10-0-140-247.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-w3-c0bbb3\": no relationship found between node 'ip-10-0-140-247.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"default-dockercfg-jlqlz\"" type="*v1.Secret"
Sep 11 12:51:30.714803 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.714852 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lkk\" (UniqueName: \"kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.714930 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.714966 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.714966 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.715031 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.714976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.733767 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.733735    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:51:30.816392 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lkk\" (UniqueName: \"kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816533 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816874 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.816954 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816859    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.817008 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.816960    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.818718 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.818698    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.818872 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.818854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:30.829933 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:30.829903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lkk\" (UniqueName: \"kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk\") pod \"rate-test-model-c0bbb3-kserve-58bdf78bdc-88265\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") " pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:32.000333 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:32.000284    2575 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 11 12:51:32.000706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:32.000376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:32.061992 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:32.061960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c0bbb3\"/\"default-dockercfg-jlqlz\""
Sep 11 12:51:32.281855 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:32.281828    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:51:32.282588 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:51:32.282551    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c13e8f5_50b9_48b2_9f6f_a824c8983704.slice/crio-5e8967b0d9581f9f4ba5f4b87cbf438619363eb4a6f1844b4b4f178c355757be WatchSource:0}: Error finding container 5e8967b0d9581f9f4ba5f4b87cbf438619363eb4a6f1844b4b4f178c355757be: Status 404 returned error can't find the container with id 5e8967b0d9581f9f4ba5f4b87cbf438619363eb4a6f1844b4b4f178c355757be
Sep 11 12:51:32.317642 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:32.317614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" event={"ID":"0c13e8f5-50b9-48b2-9f6f-a824c8983704","Type":"ContainerStarted","Data":"5e8967b0d9581f9f4ba5f4b87cbf438619363eb4a6f1844b4b4f178c355757be"}
Sep 11 12:51:33.326724 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.326689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" event={"ID":"0c13e8f5-50b9-48b2-9f6f-a824c8983704","Type":"ContainerStarted","Data":"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"}
Sep 11 12:51:33.356191 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.356132    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" podStartSLOduration=3.356095833 podStartE2EDuration="3.356095833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:51:33.354529428 +0000 UTC m=+1476.196280571" watchObservedRunningTime="2026-09-11 12:51:33.356095833 +0000 UTC m=+1476.197846958"
Sep 11 12:51:33.408640 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.408609    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"]
Sep 11 12:51:33.411933 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.411915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:33.414992 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.414974    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:51:33.415075 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.414977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:51:33.425814 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.425793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"]
Sep 11 12:51:33.440982 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.440956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4\" (UID: \"a3be93ad-485a-40b9-a014-64dec09449bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:33.542518 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.542490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4\" (UID: \"a3be93ad-485a-40b9-a014-64dec09449bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:33.543105 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.543085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4\" (UID: \"a3be93ad-485a-40b9-a014-64dec09449bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:33.722417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:33.722388    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:34.004696 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:34.004669    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"]
Sep 11 12:51:34.006456 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:51:34.006434    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3be93ad_485a_40b9_a014_64dec09449bc.slice/crio-f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1 WatchSource:0}: Error finding container f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1: Status 404 returned error can't find the container with id f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1
Sep 11 12:51:34.333357 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:34.333309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4" event={"ID":"a3be93ad-485a-40b9-a014-64dec09449bc","Type":"ContainerStarted","Data":"03b06d5f55bfea94de89bc9548e9317679f950e790dd6f701fba83362045ca55"}
Sep 11 12:51:34.333751 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:34.333376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4" event={"ID":"a3be93ad-485a-40b9-a014-64dec09449bc","Type":"ContainerStarted","Data":"f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1"}
Sep 11 12:51:34.355483 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:34.355433    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4" podStartSLOduration=1.355414253 podStartE2EDuration="1.355414253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:51:34.354526144 +0000 UTC m=+1477.196277292" watchObservedRunningTime="2026-09-11 12:51:34.355414253 +0000 UTC m=+1477.197165378"
Sep 11 12:51:35.340154 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:35.340122    2575 generic.go:356] "Generic (PLEG): container finished" podID="a3be93ad-485a-40b9-a014-64dec09449bc" containerID="03b06d5f55bfea94de89bc9548e9317679f950e790dd6f701fba83362045ca55" exitCode=0
Sep 11 12:51:35.340534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:35.340205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4" event={"ID":"a3be93ad-485a-40b9-a014-64dec09449bc","Type":"ContainerDied","Data":"03b06d5f55bfea94de89bc9548e9317679f950e790dd6f701fba83362045ca55"}
Sep 11 12:51:36.471557 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:36.471530    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:36.567590 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:36.567558    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca\") pod \"a3be93ad-485a-40b9-a014-64dec09449bc\" (UID: \"a3be93ad-485a-40b9-a014-64dec09449bc\") "
Sep 11 12:51:36.567978 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:36.567958    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3be93ad-485a-40b9-a014-64dec09449bc" (UID: "a3be93ad-485a-40b9-a014-64dec09449bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:51:36.669218 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:36.669146    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3be93ad-485a-40b9-a014-64dec09449bc-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:37.351839 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:37.351807    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"
Sep 11 12:51:37.351839 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:37.351817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4" event={"ID":"a3be93ad-485a-40b9-a014-64dec09449bc","Type":"ContainerDied","Data":"f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1"}
Sep 11 12:51:37.352067 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:37.351863    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3db52994b3db3d1cec9327be4b2c259731adede5a7586a626ae8e87543d0cf1"
Sep 11 12:51:38.632353 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.632290    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" podUID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" containerName="main" containerID="cri-o://bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd" gracePeriod=45
Sep 11 12:51:38.879159 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.879138    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:51:38.988233 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988201    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8rrs\" (UniqueName: \"kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988411 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988257    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988411 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988282    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988411 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988380    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988569 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988417    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988569 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988453    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home\") pod \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\" (UID: \"3c70d873-7491-4380-8dc5-ff5d89a6cba8\") "
Sep 11 12:51:38.988691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:51:38.988757 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:51:38.988821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988767    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home" (OuterVolumeSpecName: "home") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:51:38.988821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988789    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:38.988821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.988808    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:38.990440 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.990415    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm" (OuterVolumeSpecName: "dshm") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:51:38.990517 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.990439    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:51:38.990549 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:38.990523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs" (OuterVolumeSpecName: "kube-api-access-z8rrs") pod "3c70d873-7491-4380-8dc5-ff5d89a6cba8" (UID: "3c70d873-7491-4380-8dc5-ff5d89a6cba8"). InnerVolumeSpecName "kube-api-access-z8rrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:51:39.089905 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.089866    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rrs\" (UniqueName: \"kubernetes.io/projected/3c70d873-7491-4380-8dc5-ff5d89a6cba8-kube-api-access-z8rrs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:39.089905 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.089897    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c70d873-7491-4380-8dc5-ff5d89a6cba8-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:39.089905 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.089907    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:39.089905 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.089914    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c70d873-7491-4380-8dc5-ff5d89a6cba8-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:51:39.362098 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.362007    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" containerID="bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd" exitCode=0
Sep 11 12:51:39.362098 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.362054    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"
Sep 11 12:51:39.362318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.362093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" event={"ID":"3c70d873-7491-4380-8dc5-ff5d89a6cba8","Type":"ContainerDied","Data":"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"}
Sep 11 12:51:39.362318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.362137    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk" event={"ID":"3c70d873-7491-4380-8dc5-ff5d89a6cba8","Type":"ContainerDied","Data":"8b5c5a9f71ae56c1051a838647b2494ddcf67cd4b07dcf9c0f14d28c9610aee1"}
Sep 11 12:51:39.362318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.362154    2575 scope.go:117] "RemoveContainer" containerID="bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"
Sep 11 12:51:39.374000 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.373971    2575 scope.go:117] "RemoveContainer" containerID="bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"
Sep 11 12:51:39.374295 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:51:39.374275    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd\": container with ID starting with bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd not found: ID does not exist" containerID="bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"
Sep 11 12:51:39.374399 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.374303    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd"} err="failed to get container status \"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd\": rpc error: code = NotFound desc = could not find container \"bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd\": container with ID starting with bb35891485e8793e8889131adf5c72302cdb5e99ae2a37ee0115badeeb0884bd not found: ID does not exist"
Sep 11 12:51:39.394094 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.394065    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:51:39.403731 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.403707    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/sub-test-model-1337fa-kserve-7b6d6c87d9-nncwk"]
Sep 11 12:51:39.630891 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:39.630812    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" path="/var/lib/kubelet/pods/3c70d873-7491-4380-8dc5-ff5d89a6cba8/volumes"
Sep 11 12:51:42.000795 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:42.000762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:42.001236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:42.001043    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:42.013056 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:42.013034    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:42.388952 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:42.388863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:51:49.566045 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:49.566008    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"]
Sep 11 12:51:49.570197 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:49.570169    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e1e43-6f6z4"]
Sep 11 12:51:49.630313 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:49.630279    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3be93ad-485a-40b9-a014-64dec09449bc" path="/var/lib/kubelet/pods/a3be93ad-485a-40b9-a014-64dec09449bc/volumes"
Sep 11 12:51:57.766743 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.766712    2575 scope.go:117] "RemoveContainer" containerID="8e9d462ad589c47d0b8a7bb80abf61ea48dd58aefda016e70f27dfb0f7b13d73"
Sep 11 12:51:57.774244 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.774223    2575 scope.go:117] "RemoveContainer" containerID="073ed1f5f86d6034347c54685f0cd6367896cc83f12a331b285bd40457eecbef"
Sep 11 12:51:57.829201 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829177    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"]
Sep 11 12:51:57.829610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829594    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" containerName="main"
Sep 11 12:51:57.829610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829611    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" containerName="main"
Sep 11 12:51:57.829715 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829627    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3be93ad-485a-40b9-a014-64dec09449bc" containerName="revoke-keys"
Sep 11 12:51:57.829715 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829636    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3be93ad-485a-40b9-a014-64dec09449bc" containerName="revoke-keys"
Sep 11 12:51:57.829715 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829704    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c70d873-7491-4380-8dc5-ff5d89a6cba8" containerName="main"
Sep 11 12:51:57.829804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.829716    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3be93ad-485a-40b9-a014-64dec09449bc" containerName="revoke-keys"
Sep 11 12:51:57.834080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.834066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:51:57.837067 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.837043    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:51:57.837230 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.837219    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:51:57.844580 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.844559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"]
Sep 11 12:51:57.960371 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:57.960320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w\" (UID: \"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:51:58.061508 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:58.061455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w\" (UID: \"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:51:58.062007 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:58.061981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w\" (UID: \"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:51:58.144159 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:58.144141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:51:58.421386 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:58.421361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"]
Sep 11 12:51:58.423159 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:51:58.423132    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1b2084_4dec_4c3b_88e0_3fdb92a8a1f2.slice/crio-17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850 WatchSource:0}: Error finding container 17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850: Status 404 returned error can't find the container with id 17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850
Sep 11 12:51:58.460005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:58.459975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w" event={"ID":"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2","Type":"ContainerStarted","Data":"17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850"}
Sep 11 12:51:59.466553 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:59.466522    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" containerID="6d0bcf9a6fe39511afedd246a6379b351632bda930d1bcea348783f23d7bc221" exitCode=0
Sep 11 12:51:59.466877 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:51:59.466590    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w" event={"ID":"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2","Type":"ContainerDied","Data":"6d0bcf9a6fe39511afedd246a6379b351632bda930d1bcea348783f23d7bc221"}
Sep 11 12:52:00.612776 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:00.612752    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:52:00.780621 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:00.780562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca\") pod \"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2\" (UID: \"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2\") "
Sep 11 12:52:00.780924 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:00.780904    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" (UID: "ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:00.881252 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:00.881224    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:01.477778 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:01.477753    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"
Sep 11 12:52:01.477910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:01.477773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w" event={"ID":"ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2","Type":"ContainerDied","Data":"17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850"}
Sep 11 12:52:01.477910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:01.477821    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17042f02606693ea52fdd1ec1efb733442f3718b4d0e4aff618ae7aa9b5c3850"
Sep 11 12:52:13.534729 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.534690    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:52:13.535290 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.534947    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" podUID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" containerName="payload-processing" containerID="cri-o://a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087" gracePeriod=30
Sep 11 12:52:13.545541 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.545515    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:52:13.546217 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.546188    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" podUID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" containerName="payload-pre-processing" containerID="cri-o://1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447" gracePeriod=30
Sep 11 12:52:13.702086 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.702052    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"]
Sep 11 12:52:13.706419 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.706388    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1a4a12-qls7w"]
Sep 11 12:52:13.789417 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.789366    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:52:13.818260 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.818238    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:52:13.883499 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.883467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume\") pod \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") "
Sep 11 12:52:13.883647 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.883532    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lcht\" (UniqueName: \"kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht\") pod \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\" (UID: \"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277\") "
Sep 11 12:52:13.883869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.883830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" (UID: "ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:13.885661 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.885632    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht" (OuterVolumeSpecName: "kube-api-access-7lcht") pod "ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" (UID: "ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277"). InnerVolumeSpecName "kube-api-access-7lcht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:52:13.984284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.984249    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcwww\" (UniqueName: \"kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww\") pod \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") "
Sep 11 12:52:13.984483 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.984438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume\") pod \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\" (UID: \"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99\") "
Sep 11 12:52:13.984680 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.984664    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcht\" (UniqueName: \"kubernetes.io/projected/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-kube-api-access-7lcht\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:13.984737 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.984683    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:13.984788 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.984765    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" (UID: "ec1ce9e3-bcbb-42a1-9758-e0a30421ca99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:13.986207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:13.986188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww" (OuterVolumeSpecName: "kube-api-access-dcwww") pod "ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" (UID: "ec1ce9e3-bcbb-42a1-9758-e0a30421ca99"). InnerVolumeSpecName "kube-api-access-dcwww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:52:14.085816 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.085748    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcwww\" (UniqueName: \"kubernetes.io/projected/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-kube-api-access-dcwww\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:14.085816 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.085780    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:14.543509 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.543480    2575 generic.go:356] "Generic (PLEG): container finished" podID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" containerID="1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447" exitCode=0
Sep 11 12:52:14.543909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.543520    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"
Sep 11 12:52:14.543909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.543566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" event={"ID":"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99","Type":"ContainerDied","Data":"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"}
Sep 11 12:52:14.543909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.543630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6" event={"ID":"ec1ce9e3-bcbb-42a1-9758-e0a30421ca99","Type":"ContainerDied","Data":"cc4c35075d0d1c8008b4ba33ca7f460103cdff729e9c47cb4a7c85e6463e0335"}
Sep 11 12:52:14.543909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.543661    2575 scope.go:117] "RemoveContainer" containerID="1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"
Sep 11 12:52:14.544909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.544830    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" containerID="a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087" exitCode=0
Sep 11 12:52:14.544909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.544857    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"
Sep 11 12:52:14.545012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.544914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" event={"ID":"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277","Type":"ContainerDied","Data":"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"}
Sep 11 12:52:14.545012 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.544963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr" event={"ID":"ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277","Type":"ContainerDied","Data":"5db02737b3635e8d67bd330f00869fcd1dd592c7499674d5ef8818edfaa035c9"}
Sep 11 12:52:14.557469 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.557451    2575 scope.go:117] "RemoveContainer" containerID="1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"
Sep 11 12:52:14.557722 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:52:14.557701    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447\": container with ID starting with 1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447 not found: ID does not exist" containerID="1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"
Sep 11 12:52:14.557767 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.557729    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447"} err="failed to get container status \"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447\": rpc error: code = NotFound desc = could not find container \"1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447\": container with ID starting with 1203aa39a22d661d3e2cc2da606283c29af94d0fdb938ea62e3431487b8cf447 not found: ID does not exist"
Sep 11 12:52:14.557767 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.557743    2575 scope.go:117] "RemoveContainer" containerID="a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"
Sep 11 12:52:14.567504 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.567485    2575 scope.go:117] "RemoveContainer" containerID="a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"
Sep 11 12:52:14.568036 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:52:14.568015    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087\": container with ID starting with a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087 not found: ID does not exist" containerID="a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"
Sep 11 12:52:14.568108 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.568044    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087"} err="failed to get container status \"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087\": rpc error: code = NotFound desc = could not find container \"a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087\": container with ID starting with a9a443781b53564b7995b15b3405386cae7e88acf674ba2622857a6a97691087 not found: ID does not exist"
Sep 11 12:52:14.573105 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.573088    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:52:14.591282 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.591262    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1a4a12-7f75d87bb6-lvxt6"]
Sep 11 12:52:14.610449 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.610428    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:52:14.613402 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:14.613383    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1a4a12-df55fdb77-lbldr"]
Sep 11 12:52:15.629787 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.629756    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" path="/var/lib/kubelet/pods/ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2/volumes"
Sep 11 12:52:15.630151 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.630103    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" path="/var/lib/kubelet/pods/ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277/volumes"
Sep 11 12:52:15.630434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.630421    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" path="/var/lib/kubelet/pods/ec1ce9e3-bcbb-42a1-9758-e0a30421ca99/volumes"
Sep 11 12:52:15.697230 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697204    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:52:15.697563 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697551    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" containerName="payload-pre-processing"
Sep 11 12:52:15.697620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697564    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" containerName="payload-pre-processing"
Sep 11 12:52:15.697620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697573    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" containerName="payload-processing"
Sep 11 12:52:15.697620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697580    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" containerName="payload-processing"
Sep 11 12:52:15.697620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697600    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" containerName="revoke-keys"
Sep 11 12:52:15.697620 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697606    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" containerName="revoke-keys"
Sep 11 12:52:15.697768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697658    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae097f9c-5bb1-4b77-a3c2-b18ac5fe2277" containerName="payload-processing"
Sep 11 12:52:15.697768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697667    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1ce9e3-bcbb-42a1-9758-e0a30421ca99" containerName="payload-pre-processing"
Sep 11 12:52:15.697768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.697672    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1b2084-4dec-4c3b-88e0-3fdb92a8a1f2" containerName="revoke-keys"
Sep 11 12:52:15.702135 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.702118    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.706054 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.706027    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"rate-test-model-1337fa-kserve-self-signed-certs\""
Sep 11 12:52:15.706165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.706032    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"kube-root-ca.crt\""
Sep 11 12:52:15.706165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.706132    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"openshift-service-ca.crt\""
Sep 11 12:52:15.706165 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.706157    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1337fa\"/\"default-dockercfg-672t6\""
Sep 11 12:52:15.710615 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.710594    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:52:15.799594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9d9\" (UniqueName: \"kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.799742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.799821 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.799859 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.799906 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.799950 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.799904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.900935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901020 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.900982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9d9\" (UniqueName: \"kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.901534 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.901434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.903319 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.903297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.903507 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.903492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:15.909795 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:15.909773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9d9\" (UniqueName: \"kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9\") pod \"rate-test-model-1337fa-kserve-9b6f5c495-bct5v\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") " pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:16.012300 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:16.012270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:16.283415 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:16.283386    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:52:16.285356 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:52:16.285308    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511c6bce_0256_4773_9d1d_229a4cb3cd89.slice/crio-50a116c0d1376fc88d0f0e6e0465b1f33372f89dca610ec9dd08cee709f6bd72 WatchSource:0}: Error finding container 50a116c0d1376fc88d0f0e6e0465b1f33372f89dca610ec9dd08cee709f6bd72: Status 404 returned error can't find the container with id 50a116c0d1376fc88d0f0e6e0465b1f33372f89dca610ec9dd08cee709f6bd72
Sep 11 12:52:16.558722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:16.558629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" event={"ID":"511c6bce-0256-4773-9d1d-229a4cb3cd89","Type":"ContainerStarted","Data":"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"}
Sep 11 12:52:16.558722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:16.558682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" event={"ID":"511c6bce-0256-4773-9d1d-229a4cb3cd89","Type":"ContainerStarted","Data":"50a116c0d1376fc88d0f0e6e0465b1f33372f89dca610ec9dd08cee709f6bd72"}
Sep 11 12:52:16.577143 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:16.577102    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" podStartSLOduration=1.5770876999999999 podStartE2EDuration="1.5770877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:52:16.576717663 +0000 UTC m=+1519.418468807" watchObservedRunningTime="2026-09-11 12:52:16.5770877 +0000 UTC m=+1519.418838840"
Sep 11 12:52:20.987959 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:20.987880    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"]
Sep 11 12:52:20.991313 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:20.991292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:20.994250 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:20.994227    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:52:20.994375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:20.994283    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:52:21.000606 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.000581    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"]
Sep 11 12:52:21.044573 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.044540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn\" (UID: \"e4769d5e-346c-48a6-84b2-76bfe90076c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:21.145991 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.145956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn\" (UID: \"e4769d5e-346c-48a6-84b2-76bfe90076c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:21.146554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.146537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn\" (UID: \"e4769d5e-346c-48a6-84b2-76bfe90076c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:21.302721 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.302638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:21.572278 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.572195    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"]
Sep 11 12:52:21.575310 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:52:21.575286    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4769d5e_346c_48a6_84b2_76bfe90076c5.slice/crio-d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb WatchSource:0}: Error finding container d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb: Status 404 returned error can't find the container with id d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb
Sep 11 12:52:21.588120 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:21.588090    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn" event={"ID":"e4769d5e-346c-48a6-84b2-76bfe90076c5","Type":"ContainerStarted","Data":"d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb"}
Sep 11 12:52:22.595033 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:22.594998    2575 generic.go:356] "Generic (PLEG): container finished" podID="e4769d5e-346c-48a6-84b2-76bfe90076c5" containerID="4ae87ec1438c7fc300a638b4a3f8d17e4ad741c43a5f4d3fbae7f24686e85f9d" exitCode=0
Sep 11 12:52:22.595474 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:22.595050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn" event={"ID":"e4769d5e-346c-48a6-84b2-76bfe90076c5","Type":"ContainerDied","Data":"4ae87ec1438c7fc300a638b4a3f8d17e4ad741c43a5f4d3fbae7f24686e85f9d"}
Sep 11 12:52:23.733692 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:23.733673    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:23.768278 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:23.768260    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca\") pod \"e4769d5e-346c-48a6-84b2-76bfe90076c5\" (UID: \"e4769d5e-346c-48a6-84b2-76bfe90076c5\") "
Sep 11 12:52:23.768610 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:23.768581    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4769d5e-346c-48a6-84b2-76bfe90076c5" (UID: "e4769d5e-346c-48a6-84b2-76bfe90076c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:23.869483 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:23.869434    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769d5e-346c-48a6-84b2-76bfe90076c5-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:24.611966 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:24.611936    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"
Sep 11 12:52:24.612144 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:24.611975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn" event={"ID":"e4769d5e-346c-48a6-84b2-76bfe90076c5","Type":"ContainerDied","Data":"d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb"}
Sep 11 12:52:24.612144 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:24.612058    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8621b701295e52a66b0686d6533d059801fca992586a1d0241f0dbb85dbdccb"
Sep 11 12:52:26.013236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:26.013203    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:26.013626 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:26.013267    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:26.025402 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:26.025373    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:26.633594 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:26.633554    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:52:31.930757 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:31.930720    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:52:31.931532 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:31.931480    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" podUID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" containerName="payload-pre-processing" containerID="cri-o://5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9" gracePeriod=30
Sep 11 12:52:32.163067 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.163037    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"]
Sep 11 12:52:32.168878 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.168857    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ce2636-j9pkn"]
Sep 11 12:52:32.188560 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.188527    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:52:32.241075 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.241049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjn6s\" (UniqueName: \"kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s\") pod \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") "
Sep 11 12:52:32.241206 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.241115    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume\") pod \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\" (UID: \"7d85ba6f-e2dd-46ea-adc0-a326a18c6377\") "
Sep 11 12:52:32.241538 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.241509    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d85ba6f-e2dd-46ea-adc0-a326a18c6377" (UID: "7d85ba6f-e2dd-46ea-adc0-a326a18c6377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:32.243113 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.243090    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s" (OuterVolumeSpecName: "kube-api-access-zjn6s") pod "7d85ba6f-e2dd-46ea-adc0-a326a18c6377" (UID: "7d85ba6f-e2dd-46ea-adc0-a326a18c6377"). InnerVolumeSpecName "kube-api-access-zjn6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:52:32.342030 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.341993    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:32.342030 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.342022    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjn6s\" (UniqueName: \"kubernetes.io/projected/7d85ba6f-e2dd-46ea-adc0-a326a18c6377-kube-api-access-zjn6s\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:32.653516 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.653435    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" containerID="5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9" exitCode=0
Sep 11 12:52:32.653516 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.653477    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"
Sep 11 12:52:32.653516 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.653490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" event={"ID":"7d85ba6f-e2dd-46ea-adc0-a326a18c6377","Type":"ContainerDied","Data":"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"}
Sep 11 12:52:32.653734 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.653528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf" event={"ID":"7d85ba6f-e2dd-46ea-adc0-a326a18c6377","Type":"ContainerDied","Data":"1fec2f38337c05c0057cbf073f29308557bcbe7ba29f70192b5746fe8dfd3854"}
Sep 11 12:52:32.653734 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.653550    2575 scope.go:117] "RemoveContainer" containerID="5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"
Sep 11 12:52:32.662473 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.662360    2575 scope.go:117] "RemoveContainer" containerID="5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"
Sep 11 12:52:32.662640 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:52:32.662621    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9\": container with ID starting with 5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9 not found: ID does not exist" containerID="5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"
Sep 11 12:52:32.662697 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.662652    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9"} err="failed to get container status \"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9\": rpc error: code = NotFound desc = could not find container \"5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9\": container with ID starting with 5c0dc029dc5f8b4d56d4f4555f77c45409c659780563c118bd6e4f891dcc41a9 not found: ID does not exist"
Sep 11 12:52:32.677064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.677031    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:52:32.684309 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:32.684289    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce2636-5d4f7db67d-7zkbf"]
Sep 11 12:52:33.630104 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:33.630072    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" path="/var/lib/kubelet/pods/7d85ba6f-e2dd-46ea-adc0-a326a18c6377/volumes"
Sep 11 12:52:33.630489 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:33.630444    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4769d5e-346c-48a6-84b2-76bfe90076c5" path="/var/lib/kubelet/pods/e4769d5e-346c-48a6-84b2-76bfe90076c5/volumes"
Sep 11 12:52:39.220580 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.220539    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"]
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.220906    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4769d5e-346c-48a6-84b2-76bfe90076c5" containerName="revoke-keys"
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.220917    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4769d5e-346c-48a6-84b2-76bfe90076c5" containerName="revoke-keys"
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.220935    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" containerName="payload-pre-processing"
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.220941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" containerName="payload-pre-processing"
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.221005    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4769d5e-346c-48a6-84b2-76bfe90076c5" containerName="revoke-keys"
Sep 11 12:52:39.221060 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.221013    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d85ba6f-e2dd-46ea-adc0-a326a18c6377" containerName="payload-pre-processing"
Sep 11 12:52:39.225439 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.225415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:39.227945 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.227924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:52:39.228036 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.227968    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:52:39.233429 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.233040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"]
Sep 11 12:52:39.299434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.299399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh\" (UID: \"454838d4-b622-4c03-8339-14ef35855002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:39.400565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.400530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh\" (UID: \"454838d4-b622-4c03-8339-14ef35855002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:39.401207 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.401186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh\" (UID: \"454838d4-b622-4c03-8339-14ef35855002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:39.538665 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.538571    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:39.809565 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:39.809539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"]
Sep 11 12:52:39.811642 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:52:39.811618    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454838d4_b622_4c03_8339_14ef35855002.slice/crio-31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019 WatchSource:0}: Error finding container 31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019: Status 404 returned error can't find the container with id 31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019
Sep 11 12:52:40.693895 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:40.693860    2575 generic.go:356] "Generic (PLEG): container finished" podID="454838d4-b622-4c03-8339-14ef35855002" containerID="db39fef33b23ba2dec1a3fc8a9ea66e8f9af6408717b08eeba1740509d102be1" exitCode=0
Sep 11 12:52:40.694362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:40.693911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh" event={"ID":"454838d4-b622-4c03-8339-14ef35855002","Type":"ContainerDied","Data":"db39fef33b23ba2dec1a3fc8a9ea66e8f9af6408717b08eeba1740509d102be1"}
Sep 11 12:52:40.694362 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:40.693971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh" event={"ID":"454838d4-b622-4c03-8339-14ef35855002","Type":"ContainerStarted","Data":"31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019"}
Sep 11 12:52:41.836697 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:41.836671    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:41.929691 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:41.929659    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca\") pod \"454838d4-b622-4c03-8339-14ef35855002\" (UID: \"454838d4-b622-4c03-8339-14ef35855002\") "
Sep 11 12:52:41.930070 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:41.930041    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "454838d4-b622-4c03-8339-14ef35855002" (UID: "454838d4-b622-4c03-8339-14ef35855002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:42.031028 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:42.030941    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454838d4-b622-4c03-8339-14ef35855002-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:42.705273 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:42.705242    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"
Sep 11 12:52:42.705273 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:42.705256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh" event={"ID":"454838d4-b622-4c03-8339-14ef35855002","Type":"ContainerDied","Data":"31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019"}
Sep 11 12:52:42.705494 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:42.705308    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ecb9a3f84ef5c283ed3a5f78e0f1213ee368b570c218833c5bf6222e7f3019"
Sep 11 12:52:49.814141 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.814106    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:52:49.814683 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.814312    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" podUID="744aefc9-2242-4515-8e48-a1de92f24c02" containerName="payload-processing" containerID="cri-o://33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483" gracePeriod=30
Sep 11 12:52:49.833855 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.833827    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:52:49.834071 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.834047    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" podUID="a46a80e2-92b1-47e4-84a5-df038f274078" containerName="payload-pre-processing" containerID="cri-o://1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da" gracePeriod=30
Sep 11 12:52:49.990921 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.990888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"]
Sep 11 12:52:49.993889 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:49.993861    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5db8fe-bnggh"]
Sep 11 12:52:50.104804 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.104780    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:52:50.107584 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.107566    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:52:50.215052 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215016    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pflff\" (UniqueName: \"kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff\") pod \"744aefc9-2242-4515-8e48-a1de92f24c02\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") "
Sep 11 12:52:50.215244 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume\") pod \"a46a80e2-92b1-47e4-84a5-df038f274078\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") "
Sep 11 12:52:50.215244 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215157    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume\") pod \"744aefc9-2242-4515-8e48-a1de92f24c02\" (UID: \"744aefc9-2242-4515-8e48-a1de92f24c02\") "
Sep 11 12:52:50.215244 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215197    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k9lg\" (UniqueName: \"kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg\") pod \"a46a80e2-92b1-47e4-84a5-df038f274078\" (UID: \"a46a80e2-92b1-47e4-84a5-df038f274078\") "
Sep 11 12:52:50.215574 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a46a80e2-92b1-47e4-84a5-df038f274078" (UID: "a46a80e2-92b1-47e4-84a5-df038f274078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:50.215651 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.215545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "744aefc9-2242-4515-8e48-a1de92f24c02" (UID: "744aefc9-2242-4515-8e48-a1de92f24c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:52:50.217238 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.217211    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg" (OuterVolumeSpecName: "kube-api-access-8k9lg") pod "a46a80e2-92b1-47e4-84a5-df038f274078" (UID: "a46a80e2-92b1-47e4-84a5-df038f274078"). InnerVolumeSpecName "kube-api-access-8k9lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:52:50.217371 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.217245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff" (OuterVolumeSpecName: "kube-api-access-pflff") pod "744aefc9-2242-4515-8e48-a1de92f24c02" (UID: "744aefc9-2242-4515-8e48-a1de92f24c02"). InnerVolumeSpecName "kube-api-access-pflff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:52:50.315987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.315958    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflff\" (UniqueName: \"kubernetes.io/projected/744aefc9-2242-4515-8e48-a1de92f24c02-kube-api-access-pflff\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:50.315987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.315983    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46a80e2-92b1-47e4-84a5-df038f274078-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:50.315987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.315993    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744aefc9-2242-4515-8e48-a1de92f24c02-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:50.316192 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.316001    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k9lg\" (UniqueName: \"kubernetes.io/projected/a46a80e2-92b1-47e4-84a5-df038f274078-kube-api-access-8k9lg\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:52:50.747726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.747682    2575 generic.go:356] "Generic (PLEG): container finished" podID="744aefc9-2242-4515-8e48-a1de92f24c02" containerID="33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483" exitCode=0
Sep 11 12:52:50.747897 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.747730    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"
Sep 11 12:52:50.747897 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.747772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" event={"ID":"744aefc9-2242-4515-8e48-a1de92f24c02","Type":"ContainerDied","Data":"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"}
Sep 11 12:52:50.747897 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.747824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds" event={"ID":"744aefc9-2242-4515-8e48-a1de92f24c02","Type":"ContainerDied","Data":"d7b1b58d9033336e7edac2074a326dfb44695bd695ffc6a8ad5f03309d3c8898"}
Sep 11 12:52:50.747897 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.747846    2575 scope.go:117] "RemoveContainer" containerID="33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"
Sep 11 12:52:50.749181 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.749153    2575 generic.go:356] "Generic (PLEG): container finished" podID="a46a80e2-92b1-47e4-84a5-df038f274078" containerID="1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da" exitCode=0
Sep 11 12:52:50.749304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.749198    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"
Sep 11 12:52:50.749304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.749216    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" event={"ID":"a46a80e2-92b1-47e4-84a5-df038f274078","Type":"ContainerDied","Data":"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"}
Sep 11 12:52:50.749304 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.749258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5" event={"ID":"a46a80e2-92b1-47e4-84a5-df038f274078","Type":"ContainerDied","Data":"45c6f4759312233d4701f2e0bc9074718d4626369008a70727afee324c52de78"}
Sep 11 12:52:50.758248 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.758232    2575 scope.go:117] "RemoveContainer" containerID="33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"
Sep 11 12:52:50.758562 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:52:50.758536    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483\": container with ID starting with 33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483 not found: ID does not exist" containerID="33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"
Sep 11 12:52:50.758668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.758569    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483"} err="failed to get container status \"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483\": rpc error: code = NotFound desc = could not find container \"33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483\": container with ID starting with 33af5718d73baf536c0ec7d3ad1ff51afe84321e0cf84fa286c78e2ae4b06483 not found: ID does not exist"
Sep 11 12:52:50.758668 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.758586    2575 scope.go:117] "RemoveContainer" containerID="1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"
Sep 11 12:52:50.766303 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.766285    2575 scope.go:117] "RemoveContainer" containerID="1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"
Sep 11 12:52:50.766587 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:52:50.766570    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da\": container with ID starting with 1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da not found: ID does not exist" containerID="1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"
Sep 11 12:52:50.766636 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.766595    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da"} err="failed to get container status \"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da\": rpc error: code = NotFound desc = could not find container \"1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da\": container with ID starting with 1983fcb2cb6fba491a89f72978272423b8609927547f91a0734c6e045e8b33da not found: ID does not exist"
Sep 11 12:52:50.793838 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.793812    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:52:50.802884 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.802858    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5db8fe-6f4cc586c8-pfhh5"]
Sep 11 12:52:50.826302 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.826278    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:52:50.833158 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:50.833136    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5db8fe-5bb68f7575-frcds"]
Sep 11 12:52:51.629911 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:51.629881    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454838d4-b622-4c03-8339-14ef35855002" path="/var/lib/kubelet/pods/454838d4-b622-4c03-8339-14ef35855002/volumes"
Sep 11 12:52:51.630239 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:51.630225    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744aefc9-2242-4515-8e48-a1de92f24c02" path="/var/lib/kubelet/pods/744aefc9-2242-4515-8e48-a1de92f24c02/volumes"
Sep 11 12:52:51.630570 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:52:51.630557    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46a80e2-92b1-47e4-84a5-df038f274078" path="/var/lib/kubelet/pods/a46a80e2-92b1-47e4-84a5-df038f274078/volumes"
Sep 11 12:53:07.002733 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:07.002699    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:53:07.867112 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:07.867087    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:53:22.021909 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.021843    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" podUID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" containerName="main" containerID="cri-o://fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9" gracePeriod=45
Sep 11 12:53:22.261372 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.261348    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:53:22.309926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.309852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.309926 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.309894    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.310153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.309932    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59lkk\" (UniqueName: \"kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.310153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.309976    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.310153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310016    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.310153 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310040    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir\") pod \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\" (UID: \"0c13e8f5-50b9-48b2-9f6f-a824c8983704\") "
Sep 11 12:53:22.310375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home" (OuterVolumeSpecName: "home") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:22.310375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310319    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:22.310375 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310349    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:22.310524 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310432    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.310524 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310445    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.310524 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.310453    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.312183 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.312156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm" (OuterVolumeSpecName: "dshm") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:22.312183 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.312170    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:22.312318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.312183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk" (OuterVolumeSpecName: "kube-api-access-59lkk") pod "0c13e8f5-50b9-48b2-9f6f-a824c8983704" (UID: "0c13e8f5-50b9-48b2-9f6f-a824c8983704"). InnerVolumeSpecName "kube-api-access-59lkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:22.410893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.410860    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c13e8f5-50b9-48b2-9f6f-a824c8983704-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.410893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.410889    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59lkk\" (UniqueName: \"kubernetes.io/projected/0c13e8f5-50b9-48b2-9f6f-a824c8983704-kube-api-access-59lkk\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.410893 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.410902    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c13e8f5-50b9-48b2-9f6f-a824c8983704-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:22.885722 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.885666    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" podUID="511c6bce-0256-4773-9d1d-229a4cb3cd89" containerName="main" containerID="cri-o://8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f" gracePeriod=45
Sep 11 12:53:22.916707 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.916674    2575 generic.go:356] "Generic (PLEG): container finished" podID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" containerID="fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9" exitCode=0
Sep 11 12:53:22.916868 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.916715    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"
Sep 11 12:53:22.916868 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.916757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" event={"ID":"0c13e8f5-50b9-48b2-9f6f-a824c8983704","Type":"ContainerDied","Data":"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"}
Sep 11 12:53:22.916868 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.916795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265" event={"ID":"0c13e8f5-50b9-48b2-9f6f-a824c8983704","Type":"ContainerDied","Data":"5e8967b0d9581f9f4ba5f4b87cbf438619363eb4a6f1844b4b4f178c355757be"}
Sep 11 12:53:22.916868 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.916812    2575 scope.go:117] "RemoveContainer" containerID="fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"
Sep 11 12:53:22.944555 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.944523    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:53:22.948345 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.948310    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c0bbb3/rate-test-model-c0bbb3-kserve-58bdf78bdc-88265"]
Sep 11 12:53:22.996381 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.996359    2575 scope.go:117] "RemoveContainer" containerID="fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"
Sep 11 12:53:22.996786 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:53:22.996763    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9\": container with ID starting with fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9 not found: ID does not exist" containerID="fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"
Sep 11 12:53:22.996850 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:22.996801    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9"} err="failed to get container status \"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9\": rpc error: code = NotFound desc = could not find container \"fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9\": container with ID starting with fdbfd9b8b1b2320cb59195233c7eae8ecec0ec4023b1182492753c08e1d3c6d9 not found: ID does not exist"
Sep 11 12:53:23.132578 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.132553    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:53:23.217281 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217247    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217291    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9d9\" (UniqueName: \"kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217358    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217394    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217464 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217441    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217483    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs\") pod \"511c6bce-0256-4773-9d1d-229a4cb3cd89\" (UID: \"511c6bce-0256-4773-9d1d-229a4cb3cd89\") "
Sep 11 12:53:23.217706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217589    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home" (OuterVolumeSpecName: "home") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:23.217706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache" (OuterVolumeSpecName: "model-cache") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:23.217847 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217735    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:23.217847 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217776    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.217847 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.217794    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.219471 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.219449    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm" (OuterVolumeSpecName: "dshm") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:23.219471 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.219458    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:23.219574 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.219510    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9" (OuterVolumeSpecName: "kube-api-access-ft9d9") pod "511c6bce-0256-4773-9d1d-229a4cb3cd89" (UID: "511c6bce-0256-4773-9d1d-229a4cb3cd89"). InnerVolumeSpecName "kube-api-access-ft9d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:23.319188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.319150    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9d9\" (UniqueName: \"kubernetes.io/projected/511c6bce-0256-4773-9d1d-229a4cb3cd89-kube-api-access-ft9d9\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.319188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.319182    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.319188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.319192    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511c6bce-0256-4773-9d1d-229a4cb3cd89-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.319188 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.319200    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511c6bce-0256-4773-9d1d-229a4cb3cd89-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:53:23.629858 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.629779    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" path="/var/lib/kubelet/pods/0c13e8f5-50b9-48b2-9f6f-a824c8983704/volumes"
Sep 11 12:53:23.923170 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.923080    2575 generic.go:356] "Generic (PLEG): container finished" podID="511c6bce-0256-4773-9d1d-229a4cb3cd89" containerID="8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f" exitCode=0
Sep 11 12:53:23.923170 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.923142    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"
Sep 11 12:53:23.923434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.923168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" event={"ID":"511c6bce-0256-4773-9d1d-229a4cb3cd89","Type":"ContainerDied","Data":"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"}
Sep 11 12:53:23.923434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.923221    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v" event={"ID":"511c6bce-0256-4773-9d1d-229a4cb3cd89","Type":"ContainerDied","Data":"50a116c0d1376fc88d0f0e6e0465b1f33372f89dca610ec9dd08cee709f6bd72"}
Sep 11 12:53:23.923434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.923242    2575 scope.go:117] "RemoveContainer" containerID="8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"
Sep 11 12:53:23.931246 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.931227    2575 scope.go:117] "RemoveContainer" containerID="8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"
Sep 11 12:53:23.931535 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:53:23.931516    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f\": container with ID starting with 8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f not found: ID does not exist" containerID="8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"
Sep 11 12:53:23.931596 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.931543    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f"} err="failed to get container status \"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f\": rpc error: code = NotFound desc = could not find container \"8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f\": container with ID starting with 8cb5bd801bb139aa3c0aca31b078e7f7f02cf23d4e0d79ad1f42558abbfbfa8f not found: ID does not exist"
Sep 11 12:53:23.940293 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.940268    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:53:23.946200 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:23.946178    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1337fa/rate-test-model-1337fa-kserve-9b6f5c495-bct5v"]
Sep 11 12:53:25.629902 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:25.629869    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511c6bce-0256-4773-9d1d-229a4cb3cd89" path="/var/lib/kubelet/pods/511c6bce-0256-4773-9d1d-229a4cb3cd89/volumes"
Sep 11 12:53:43.111083 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111054    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111427    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744aefc9-2242-4515-8e48-a1de92f24c02" containerName="payload-processing"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111441    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="744aefc9-2242-4515-8e48-a1de92f24c02" containerName="payload-processing"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111461    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454838d4-b622-4c03-8339-14ef35855002" containerName="revoke-keys"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111468    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="454838d4-b622-4c03-8339-14ef35855002" containerName="revoke-keys"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111476    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46a80e2-92b1-47e4-84a5-df038f274078" containerName="payload-pre-processing"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111482    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46a80e2-92b1-47e4-84a5-df038f274078" containerName="payload-pre-processing"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111493    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" containerName="main"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111498    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" containerName="main"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111505    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c6bce-0256-4773-9d1d-229a4cb3cd89" containerName="main"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111509    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c6bce-0256-4773-9d1d-229a4cb3cd89" containerName="main"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111571    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="454838d4-b622-4c03-8339-14ef35855002" containerName="revoke-keys"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111581    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="744aefc9-2242-4515-8e48-a1de92f24c02" containerName="payload-processing"
Sep 11 12:53:43.111582 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111589    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c13e8f5-50b9-48b2-9f6f-a824c8983704" containerName="main"
Sep 11 12:53:43.111986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111595    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46a80e2-92b1-47e4-84a5-df038f274078" containerName="payload-pre-processing"
Sep 11 12:53:43.111986 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.111601    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c6bce-0256-4773-9d1d-229a4cb3cd89" containerName="main"
Sep 11 12:53:43.114308 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.114293    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.117089 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.117071    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c5dca3-dockercfg-gqhrm\""
Sep 11 12:53:43.117196 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.117177    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c5dca3\""
Sep 11 12:53:43.130796 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.130776    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:53:43.173001 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.172975    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:53:43.175910 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.175894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.188320 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.188299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:53:43.293735 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.293692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxp7\" (UniqueName: \"kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.293898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.293747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl8w\" (UniqueName: \"kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.293898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.293767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.293898 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.293811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.394446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.394371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxp7\" (UniqueName: \"kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.394446 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.394434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl8w\" (UniqueName: \"kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.394600 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.394460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.394600 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.394503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.395133 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.395113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.395189 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.395172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.403776 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.403753    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl8w\" (UniqueName: \"kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w\") pod \"payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.404179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.404159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxp7\" (UniqueName: \"kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7\") pod \"payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.424276 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.424254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:43.485922 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.485754    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:43.699847 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.699820    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:53:43.701976 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:53:43.701947    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25f907e_11b9_447d_8703_0ac2a8c42778.slice/crio-3e20110f55ebf2117dc51a24ec938c2170c26da8d379d3b466dac0fd66fd76bf WatchSource:0}: Error finding container 3e20110f55ebf2117dc51a24ec938c2170c26da8d379d3b466dac0fd66fd76bf: Status 404 returned error can't find the container with id 3e20110f55ebf2117dc51a24ec938c2170c26da8d379d3b466dac0fd66fd76bf
Sep 11 12:53:43.760865 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:43.760840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:53:43.763268 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:53:43.763204    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68abb0f_6ccd_4dbb_a748_79ae0f3d428c.slice/crio-f18a826d568ee5d4c43352cd81b67b64f87494c04a05552ba92aecd083b54810 WatchSource:0}: Error finding container f18a826d568ee5d4c43352cd81b67b64f87494c04a05552ba92aecd083b54810: Status 404 returned error can't find the container with id f18a826d568ee5d4c43352cd81b67b64f87494c04a05552ba92aecd083b54810
Sep 11 12:53:44.024192 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.024094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" event={"ID":"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c","Type":"ContainerStarted","Data":"a145083aa7f4b25576379ce96619ea922c30dc76b8fb490947eb667e68e5e52f"}
Sep 11 12:53:44.024192 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.024161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:44.024192 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.024184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" event={"ID":"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c","Type":"ContainerStarted","Data":"f18a826d568ee5d4c43352cd81b67b64f87494c04a05552ba92aecd083b54810"}
Sep 11 12:53:44.025629 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.025600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" event={"ID":"a25f907e-11b9-447d-8703-0ac2a8c42778","Type":"ContainerStarted","Data":"5b2c2b75398c354d3c037d4c2d4d9a9f5ffde20abed89b7918fdd279217f1e24"}
Sep 11 12:53:44.025750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.025641    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" event={"ID":"a25f907e-11b9-447d-8703-0ac2a8c42778","Type":"ContainerStarted","Data":"3e20110f55ebf2117dc51a24ec938c2170c26da8d379d3b466dac0fd66fd76bf"}
Sep 11 12:53:44.025750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.025714    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:44.041689 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.041646    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" podStartSLOduration=1.041633159 podStartE2EDuration="1.041633159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:53:44.039879999 +0000 UTC m=+1606.881631134" watchObservedRunningTime="2026-09-11 12:53:44.041633159 +0000 UTC m=+1606.883384284"
Sep 11 12:53:44.057781 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:44.057745    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" podStartSLOduration=1.057730624 podStartE2EDuration="1.057730624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:53:44.055623026 +0000 UTC m=+1606.897374169" watchObservedRunningTime="2026-09-11 12:53:44.057730624 +0000 UTC m=+1606.899481756"
Sep 11 12:53:55.033475 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:55.033403    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:53:55.033843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:55.033478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:53:57.825746 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:53:57.825712    2575 scope.go:117] "RemoveContainer" containerID="6792b2e4bf8ef4c2344b8e83e42c9e27b32fafaf992359a5d98f90e204405cd4"
Sep 11 12:54:11.339064 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:11.339031    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 12:54:11.456143 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:11.456118    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg_f68abb0f-6ccd-4dbb-a748-79ae0f3d428c/payload-processing/0.log"
Sep 11 12:54:16.881570 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.881537    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:54:16.884980 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.884962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.887652 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.887631    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c689a2\"/\"kube-root-ca.crt\""
Sep 11 12:54:16.887773 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.887670    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c689a2\"/\"openshift-service-ca.crt\""
Sep 11 12:54:16.888971 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.888951    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c689a2\"/\"ipp-route-c689a2-kserve-self-signed-certs\""
Sep 11 12:54:16.889066 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.889024    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c689a2\"/\"default-dockercfg-mcfp4\""
Sep 11 12:54:16.896455 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.896432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:54:16.986410 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.986618 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.986618 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49grw\" (UniqueName: \"kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.986618 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.986618 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:16.986793 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:16.986640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.087999 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.087967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49grw\" (UniqueName: \"kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088179 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088496 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088598 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.088598 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.088533    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.090434 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.090408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.090670 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.090653    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.096312 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.096290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49grw\" (UniqueName: \"kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw\") pod \"ipp-route-c689a2-kserve-84dfc744c-894mx\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") " pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.195612 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.195579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:17.473560 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:17.473471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:54:17.478585 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:54:17.478553    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32ee2da9_1f7d_4504_a1ae_010e000025e8.slice/crio-ccf5897fe91b83178aa3c01228a799f3afee3dfa2847fa2a5089244e9ffe01bf WatchSource:0}: Error finding container ccf5897fe91b83178aa3c01228a799f3afee3dfa2847fa2a5089244e9ffe01bf: Status 404 returned error can't find the container with id ccf5897fe91b83178aa3c01228a799f3afee3dfa2847fa2a5089244e9ffe01bf
Sep 11 12:54:18.204662 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:18.204616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" event={"ID":"32ee2da9-1f7d-4504-a1ae-010e000025e8","Type":"ContainerStarted","Data":"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"}
Sep 11 12:54:18.205071 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:18.204675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" event={"ID":"32ee2da9-1f7d-4504-a1ae-010e000025e8","Type":"ContainerStarted","Data":"ccf5897fe91b83178aa3c01228a799f3afee3dfa2847fa2a5089244e9ffe01bf"}
Sep 11 12:54:18.223933 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:18.223883    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" podStartSLOduration=2.223864882 podStartE2EDuration="2.223864882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:54:18.223147917 +0000 UTC m=+1641.064899043" watchObservedRunningTime="2026-09-11 12:54:18.223864882 +0000 UTC m=+1641.065616007"
Sep 11 12:54:27.195855 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:27.195813    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:27.195855 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:27.195867    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:27.208844 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:27.208815    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:27.257904 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:27.257875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:54:42.045578 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:42.045540    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 12:54:42.163442 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:42.163411    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg_f68abb0f-6ccd-4dbb-a748-79ae0f3d428c/payload-processing/0.log"
Sep 11 12:54:57.850878 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:54:57.850846    2575 scope.go:117] "RemoveContainer" containerID="b30514e9bc2b9fc9ae41fb8a1645384b4eb2c2874bc7814e6bff86ab79f05b51"
Sep 11 12:55:11.413223 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.413191    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"]
Sep 11 12:55:11.416589 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.416571    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:11.419398 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.419375    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:55:11.419508 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.419380    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:55:11.425460 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.425437    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"]
Sep 11 12:55:11.558807 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.558773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs\" (UID: \"a5d1da60-8892-4f13-a9fd-6186a670f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:11.659943 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.659891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs\" (UID: \"a5d1da60-8892-4f13-a9fd-6186a670f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:11.660514 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.660494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs\" (UID: \"a5d1da60-8892-4f13-a9fd-6186a670f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:11.727818 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.727797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:11.996603 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:11.996577    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"]
Sep 11 12:55:11.998523 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:55:11.998496    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d1da60_8892_4f13_a9fd_6186a670f5a4.slice/crio-35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c WatchSource:0}: Error finding container 35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c: Status 404 returned error can't find the container with id 35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c
Sep 11 12:55:12.476370 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:12.476313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs" event={"ID":"a5d1da60-8892-4f13-a9fd-6186a670f5a4","Type":"ContainerStarted","Data":"be3b7e4e7c7884258e561c92acf431a88851221cf41ab649dc24fd053d88e496"}
Sep 11 12:55:12.476679 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:12.476386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs" event={"ID":"a5d1da60-8892-4f13-a9fd-6186a670f5a4","Type":"ContainerStarted","Data":"35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c"}
Sep 11 12:55:12.493236 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:12.493190    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs" podStartSLOduration=1.4931739689999999 podStartE2EDuration="1.493173969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:55:12.492949527 +0000 UTC m=+1695.334700655" watchObservedRunningTime="2026-09-11 12:55:12.493173969 +0000 UTC m=+1695.334925095"
Sep 11 12:55:13.482338 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:13.482291    2575 generic.go:356] "Generic (PLEG): container finished" podID="a5d1da60-8892-4f13-a9fd-6186a670f5a4" containerID="be3b7e4e7c7884258e561c92acf431a88851221cf41ab649dc24fd053d88e496" exitCode=0
Sep 11 12:55:13.482705 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:13.482376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs" event={"ID":"a5d1da60-8892-4f13-a9fd-6186a670f5a4","Type":"ContainerDied","Data":"be3b7e4e7c7884258e561c92acf431a88851221cf41ab649dc24fd053d88e496"}
Sep 11 12:55:14.618728 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:14.618696    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:14.786293 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:14.786216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca\") pod \"a5d1da60-8892-4f13-a9fd-6186a670f5a4\" (UID: \"a5d1da60-8892-4f13-a9fd-6186a670f5a4\") "
Sep 11 12:55:14.786602 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:14.786579    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5d1da60-8892-4f13-a9fd-6186a670f5a4" (UID: "a5d1da60-8892-4f13-a9fd-6186a670f5a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:55:14.887706 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:14.887678    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d1da60-8892-4f13-a9fd-6186a670f5a4-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:55:15.493806 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:15.493771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs" event={"ID":"a5d1da60-8892-4f13-a9fd-6186a670f5a4","Type":"ContainerDied","Data":"35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c"}
Sep 11 12:55:15.494014 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:15.493827    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f3cba0fc8df0eb50f6c6f7ae5b8a52765bae80002590571f3ab1af52ee657c"
Sep 11 12:55:15.494014 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:15.493780    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"
Sep 11 12:55:17.756239 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:17.756206    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"]
Sep 11 12:55:17.760021 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:17.760000    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-96a7d1-qj8bs"]
Sep 11 12:55:19.630049 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:19.630013    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d1da60-8892-4f13-a9fd-6186a670f5a4" path="/var/lib/kubelet/pods/a5d1da60-8892-4f13-a9fd-6186a670f5a4/volumes"
Sep 11 12:55:28.061912 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.061834    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"]
Sep 11 12:55:28.062310 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.062190    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d1da60-8892-4f13-a9fd-6186a670f5a4" containerName="revoke-keys"
Sep 11 12:55:28.062310 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.062201    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d1da60-8892-4f13-a9fd-6186a670f5a4" containerName="revoke-keys"
Sep 11 12:55:28.062432 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.062293    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d1da60-8892-4f13-a9fd-6186a670f5a4" containerName="revoke-keys"
Sep 11 12:55:28.065547 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.065529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:28.068045 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.068023    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:55:28.068164 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.068103    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:55:28.072371 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.072346    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"]
Sep 11 12:55:28.102682 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.102656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k\" (UID: \"823815cd-a619-4a73-a483-ec7392f21e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:28.203984 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.203953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k\" (UID: \"823815cd-a619-4a73-a483-ec7392f21e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:28.204592 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.204569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k\" (UID: \"823815cd-a619-4a73-a483-ec7392f21e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:28.378216 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.378143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:28.649061 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:28.648977    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"]
Sep 11 12:55:28.652701 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:55:28.652674    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823815cd_a619_4a73_a483_ec7392f21e84.slice/crio-9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13 WatchSource:0}: Error finding container 9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13: Status 404 returned error can't find the container with id 9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13
Sep 11 12:55:29.563647 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:29.563592    2575 generic.go:356] "Generic (PLEG): container finished" podID="823815cd-a619-4a73-a483-ec7392f21e84" containerID="2c709a0c2194d0afd8f61052e437fecaedab30eb86c0e476b115879f75c13a7e" exitCode=0
Sep 11 12:55:29.563978 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:29.563676    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k" event={"ID":"823815cd-a619-4a73-a483-ec7392f21e84","Type":"ContainerDied","Data":"2c709a0c2194d0afd8f61052e437fecaedab30eb86c0e476b115879f75c13a7e"}
Sep 11 12:55:29.563978 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:29.563728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k" event={"ID":"823815cd-a619-4a73-a483-ec7392f21e84","Type":"ContainerStarted","Data":"9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13"}
Sep 11 12:55:30.697105 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:30.697079    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:30.727087 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:30.727063    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca\") pod \"823815cd-a619-4a73-a483-ec7392f21e84\" (UID: \"823815cd-a619-4a73-a483-ec7392f21e84\") "
Sep 11 12:55:30.727497 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:30.727472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "823815cd-a619-4a73-a483-ec7392f21e84" (UID: "823815cd-a619-4a73-a483-ec7392f21e84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:55:30.828109 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:30.828052    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823815cd-a619-4a73-a483-ec7392f21e84-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:55:31.575050 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:31.575017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"
Sep 11 12:55:31.575211 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:31.575053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k" event={"ID":"823815cd-a619-4a73-a483-ec7392f21e84","Type":"ContainerDied","Data":"9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13"}
Sep 11 12:55:31.575211 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:31.575093    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dff4d51d12d431538de6bdb34b7407929139f6d384cf7cb3587fb6e2748dc13"
Sep 11 12:55:43.967005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:43.966968    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"]
Sep 11 12:55:43.969674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:43.969646    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c689a2-tvm4k"]
Sep 11 12:55:45.631342 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:45.631289    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823815cd-a619-4a73-a483-ec7392f21e84" path="/var/lib/kubelet/pods/823815cd-a619-4a73-a483-ec7392f21e84/volumes"
Sep 11 12:55:49.312955 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:55:49.312911    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:56:04.332387 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.332305    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" podUID="32ee2da9-1f7d-4504-a1ae-010e000025e8" containerName="main" containerID="cri-o://0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13" gracePeriod=45
Sep 11 12:56:04.582013 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.581983    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:56:04.720437 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720410    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720445    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49grw\" (UniqueName: \"kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720506    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720605 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs\") pod \"32ee2da9-1f7d-4504-a1ae-010e000025e8\" (UID: \"32ee2da9-1f7d-4504-a1ae-010e000025e8\") "
Sep 11 12:56:04.720869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:56:04.720869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home" (OuterVolumeSpecName: "home") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:56:04.720869 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:56:04.720985 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720901    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:04.720985 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720915    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:04.720985 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.720931    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:04.722593 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.722571    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm" (OuterVolumeSpecName: "dshm") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:56:04.722697 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.722605    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:56:04.722697 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.722611    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw" (OuterVolumeSpecName: "kube-api-access-49grw") pod "32ee2da9-1f7d-4504-a1ae-010e000025e8" (UID: "32ee2da9-1f7d-4504-a1ae-010e000025e8"). InnerVolumeSpecName "kube-api-access-49grw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:56:04.751258 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.751235    2575 generic.go:356] "Generic (PLEG): container finished" podID="32ee2da9-1f7d-4504-a1ae-010e000025e8" containerID="0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13" exitCode=0
Sep 11 12:56:04.751363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.751274    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"
Sep 11 12:56:04.751363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.751285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" event={"ID":"32ee2da9-1f7d-4504-a1ae-010e000025e8","Type":"ContainerDied","Data":"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"}
Sep 11 12:56:04.751363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.751315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx" event={"ID":"32ee2da9-1f7d-4504-a1ae-010e000025e8","Type":"ContainerDied","Data":"ccf5897fe91b83178aa3c01228a799f3afee3dfa2847fa2a5089244e9ffe01bf"}
Sep 11 12:56:04.751363 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.751351    2575 scope.go:117] "RemoveContainer" containerID="0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"
Sep 11 12:56:04.759718 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.759697    2575 scope.go:117] "RemoveContainer" containerID="0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"
Sep 11 12:56:04.759998 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:56:04.759980    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13\": container with ID starting with 0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13 not found: ID does not exist" containerID="0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"
Sep 11 12:56:04.760068 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.760006    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13"} err="failed to get container status \"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13\": rpc error: code = NotFound desc = could not find container \"0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13\": container with ID starting with 0c5211dd910854077d130e72c398769374164cad99577acd45eebfd10525cd13 not found: ID does not exist"
Sep 11 12:56:04.772474 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.772447    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:56:04.778554 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.778534    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c689a2/ipp-route-c689a2-kserve-84dfc744c-894mx"]
Sep 11 12:56:04.822176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.822157    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ee2da9-1f7d-4504-a1ae-010e000025e8-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:04.822176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.822175    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49grw\" (UniqueName: \"kubernetes.io/projected/32ee2da9-1f7d-4504-a1ae-010e000025e8-kube-api-access-49grw\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:04.822288 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:04.822185    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ee2da9-1f7d-4504-a1ae-010e000025e8-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:05.630203 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:05.630171    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ee2da9-1f7d-4504-a1ae-010e000025e8" path="/var/lib/kubelet/pods/32ee2da9-1f7d-4504-a1ae-010e000025e8/volumes"
Sep 11 12:56:50.243875 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.243795    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"]
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244160    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823815cd-a619-4a73-a483-ec7392f21e84" containerName="revoke-keys"
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244171    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="823815cd-a619-4a73-a483-ec7392f21e84" containerName="revoke-keys"
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244190    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ee2da9-1f7d-4504-a1ae-010e000025e8" containerName="main"
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244195    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ee2da9-1f7d-4504-a1ae-010e000025e8" containerName="main"
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244256    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="823815cd-a619-4a73-a483-ec7392f21e84" containerName="revoke-keys"
Sep 11 12:56:50.244277 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.244264    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ee2da9-1f7d-4504-a1ae-010e000025e8" containerName="main"
Sep 11 12:56:50.247109 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.247092    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:50.249660 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.249634    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ww7tv\""
Sep 11 12:56:50.249785 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.249665    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:56:50.266217 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.266184    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"]
Sep 11 12:56:50.397593 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.397553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd\" (UID: \"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:50.499099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.499012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd\" (UID: \"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:50.499803 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.499781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd\" (UID: \"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:50.557119 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.557098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:50.829765 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.829740    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"]
Sep 11 12:56:50.831576 ip-10-0-140-247 kubenswrapper[2575]: W0911 12:56:50.831547    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9f448d_5d4b_452e_8e09_8ebb4a2e9214.slice/crio-b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb WatchSource:0}: Error finding container b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb: Status 404 returned error can't find the container with id b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb
Sep 11 12:56:50.985133 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:50.985100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd" event={"ID":"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214","Type":"ContainerStarted","Data":"b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb"}
Sep 11 12:56:51.991811 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:51.991780    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" containerID="adc246cdb6c27b775167f3f718556af604bfcc1867dfe1c7de90c83d227cd76e" exitCode=0
Sep 11 12:56:51.992208 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:51.991866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd" event={"ID":"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214","Type":"ContainerDied","Data":"adc246cdb6c27b775167f3f718556af604bfcc1867dfe1c7de90c83d227cd76e"}
Sep 11 12:56:53.127888 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:53.127861    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:53.223284 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:53.223261    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca\") pod \"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214\" (UID: \"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214\") "
Sep 11 12:56:53.223597 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:53.223575    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" (UID: "5c9f448d-5d4b-452e-8e09-8ebb4a2e9214"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:56:53.323852 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:53.323795    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214-maas-api-service-ca\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:56:54.002686 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:54.002657    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"
Sep 11 12:56:54.002834 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:54.002691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd" event={"ID":"5c9f448d-5d4b-452e-8e09-8ebb4a2e9214","Type":"ContainerDied","Data":"b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb"}
Sep 11 12:56:54.002834 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:56:54.002735    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19fdfa1eae835044af9a978e7bddd36151d3397ecb305dae5db3a21b6ae22bb"
Sep 11 12:57:00.855429 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:00.855396    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:57:00.855842 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:00.855604    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" podUID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" containerName="payload-processing" containerID="cri-o://a145083aa7f4b25576379ce96619ea922c30dc76b8fb490947eb667e68e5e52f" gracePeriod=30
Sep 11 12:57:00.861901 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:00.861873    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:57:00.862348 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:00.862304    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" podUID="a25f907e-11b9-447d-8703-0ac2a8c42778" containerName="payload-pre-processing" containerID="cri-o://5b2c2b75398c354d3c037d4c2d4d9a9f5ffde20abed89b7918fdd279217f1e24" gracePeriod=30
Sep 11 12:57:01.012890 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.012841    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"]
Sep 11 12:57:01.017192 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.017160    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5dca3-mqckd"]
Sep 11 12:57:01.040242 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.040178    2575 generic.go:356] "Generic (PLEG): container finished" podID="a25f907e-11b9-447d-8703-0ac2a8c42778" containerID="5b2c2b75398c354d3c037d4c2d4d9a9f5ffde20abed89b7918fdd279217f1e24" exitCode=0
Sep 11 12:57:01.040429 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.040264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" event={"ID":"a25f907e-11b9-447d-8703-0ac2a8c42778","Type":"ContainerDied","Data":"5b2c2b75398c354d3c037d4c2d4d9a9f5ffde20abed89b7918fdd279217f1e24"}
Sep 11 12:57:01.042307 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.042278    2575 generic.go:356] "Generic (PLEG): container finished" podID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" containerID="a145083aa7f4b25576379ce96619ea922c30dc76b8fb490947eb667e68e5e52f" exitCode=0
Sep 11 12:57:01.042418 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.042352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" event={"ID":"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c","Type":"ContainerDied","Data":"a145083aa7f4b25576379ce96619ea922c30dc76b8fb490947eb667e68e5e52f"}
Sep 11 12:57:01.147176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.147149    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:57:01.150066 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.150048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:57:01.286938 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.286900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brxp7\" (UniqueName: \"kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7\") pod \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") "
Sep 11 12:57:01.286938 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.286938    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume\") pod \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\" (UID: \"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c\") "
Sep 11 12:57:01.287185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.286973    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnl8w\" (UniqueName: \"kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w\") pod \"a25f907e-11b9-447d-8703-0ac2a8c42778\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") "
Sep 11 12:57:01.287185 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.287087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume\") pod \"a25f907e-11b9-447d-8703-0ac2a8c42778\" (UID: \"a25f907e-11b9-447d-8703-0ac2a8c42778\") "
Sep 11 12:57:01.287457 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.287426    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" (UID: "f68abb0f-6ccd-4dbb-a748-79ae0f3d428c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:57:01.287550 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.287464    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25f907e-11b9-447d-8703-0ac2a8c42778" (UID: "a25f907e-11b9-447d-8703-0ac2a8c42778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:57:01.288993 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.288973    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w" (OuterVolumeSpecName: "kube-api-access-qnl8w") pod "a25f907e-11b9-447d-8703-0ac2a8c42778" (UID: "a25f907e-11b9-447d-8703-0ac2a8c42778"). InnerVolumeSpecName "kube-api-access-qnl8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:57:01.289099 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.288984    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7" (OuterVolumeSpecName: "kube-api-access-brxp7") pod "f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" (UID: "f68abb0f-6ccd-4dbb-a748-79ae0f3d428c"). InnerVolumeSpecName "kube-api-access-brxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:57:01.388525 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.388445    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25f907e-11b9-447d-8703-0ac2a8c42778-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:01.388525 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.388473    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxp7\" (UniqueName: \"kubernetes.io/projected/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-kube-api-access-brxp7\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:01.388525 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.388484    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:01.388525 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.388493    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnl8w\" (UniqueName: \"kubernetes.io/projected/a25f907e-11b9-447d-8703-0ac2a8c42778-kube-api-access-qnl8w\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:01.630881 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:01.630845    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" path="/var/lib/kubelet/pods/5c9f448d-5d4b-452e-8e09-8ebb4a2e9214/volumes"
Sep 11 12:57:02.049295 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.049266    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"
Sep 11 12:57:02.049763 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.049300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9" event={"ID":"a25f907e-11b9-447d-8703-0ac2a8c42778","Type":"ContainerDied","Data":"3e20110f55ebf2117dc51a24ec938c2170c26da8d379d3b466dac0fd66fd76bf"}
Sep 11 12:57:02.049763 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.049384    2575 scope.go:117] "RemoveContainer" containerID="5b2c2b75398c354d3c037d4c2d4d9a9f5ffde20abed89b7918fdd279217f1e24"
Sep 11 12:57:02.050785 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.050764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg" event={"ID":"f68abb0f-6ccd-4dbb-a748-79ae0f3d428c","Type":"ContainerDied","Data":"f18a826d568ee5d4c43352cd81b67b64f87494c04a05552ba92aecd083b54810"}
Sep 11 12:57:02.050843 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.050770    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"
Sep 11 12:57:02.058005 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.057981    2575 scope.go:117] "RemoveContainer" containerID="a145083aa7f4b25576379ce96619ea922c30dc76b8fb490947eb667e68e5e52f"
Sep 11 12:57:02.069251 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.069232    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:57:02.074343 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.074310    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5dca3-5ccf9dcfc4-xjqk9"]
Sep 11 12:57:02.084231 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.084212    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:57:02.087793 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:02.087774    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5dca3-845f9566d9-wxqgg"]
Sep 11 12:57:03.630962 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:03.630926    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25f907e-11b9-447d-8703-0ac2a8c42778" path="/var/lib/kubelet/pods/a25f907e-11b9-447d-8703-0ac2a8c42778/volumes"
Sep 11 12:57:03.631374 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:03.631282    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" path="/var/lib/kubelet/pods/f68abb0f-6ccd-4dbb-a748-79ae0f3d428c/volumes"
Sep 11 12:57:18.365303 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.365264    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:57:18.365937 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.365511    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" podUID="2ee1da40-14f1-4405-b691-42dcbb7c5552" containerName="payload-processing" containerID="cri-o://d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b" gracePeriod=30
Sep 11 12:57:18.608457 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.608428    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:57:18.635604 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.635523    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56gfb\" (UniqueName: \"kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb\") pod \"2ee1da40-14f1-4405-b691-42dcbb7c5552\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") "
Sep 11 12:57:18.635775 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.635611    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume\") pod \"2ee1da40-14f1-4405-b691-42dcbb7c5552\" (UID: \"2ee1da40-14f1-4405-b691-42dcbb7c5552\") "
Sep 11 12:57:18.636176 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.636122    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ee1da40-14f1-4405-b691-42dcbb7c5552" (UID: "2ee1da40-14f1-4405-b691-42dcbb7c5552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:57:18.637754 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.637726    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb" (OuterVolumeSpecName: "kube-api-access-56gfb") pod "2ee1da40-14f1-4405-b691-42dcbb7c5552" (UID: "2ee1da40-14f1-4405-b691-42dcbb7c5552"). InnerVolumeSpecName "kube-api-access-56gfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:57:18.736611 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.736567    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gfb\" (UniqueName: \"kubernetes.io/projected/2ee1da40-14f1-4405-b691-42dcbb7c5552-kube-api-access-56gfb\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:18.736611 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:18.736602    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1da40-14f1-4405-b691-42dcbb7c5552-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:19.137608 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.137573    2575 generic.go:356] "Generic (PLEG): container finished" podID="2ee1da40-14f1-4405-b691-42dcbb7c5552" containerID="d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b" exitCode=0
Sep 11 12:57:19.137808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.137623    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"
Sep 11 12:57:19.137808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.137656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" event={"ID":"2ee1da40-14f1-4405-b691-42dcbb7c5552","Type":"ContainerDied","Data":"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"}
Sep 11 12:57:19.137808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.137721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927" event={"ID":"2ee1da40-14f1-4405-b691-42dcbb7c5552","Type":"ContainerDied","Data":"725b478386fc7e12237aeffcfd3da8e813444e1f1684a7ba3d23912de9ee53f8"}
Sep 11 12:57:19.137808 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.137752    2575 scope.go:117] "RemoveContainer" containerID="d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"
Sep 11 12:57:19.146225 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.146208    2575 scope.go:117] "RemoveContainer" containerID="d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"
Sep 11 12:57:19.146549 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:57:19.146531    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b\": container with ID starting with d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b not found: ID does not exist" containerID="d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"
Sep 11 12:57:19.146608 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.146558    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b"} err="failed to get container status \"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b\": rpc error: code = NotFound desc = could not find container \"d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b\": container with ID starting with d4baf109e63750fba051bd2ca0cd3e4a7a48410d67f1ff8af76e93d9b968ba3b not found: ID does not exist"
Sep 11 12:57:19.159768 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.159739    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:57:19.164525 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.164504    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c0bbb3-6f9cd65b87-49927"]
Sep 11 12:57:19.629772 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:19.629740    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee1da40-14f1-4405-b691-42dcbb7c5552" path="/var/lib/kubelet/pods/2ee1da40-14f1-4405-b691-42dcbb7c5552/volumes"
Sep 11 12:57:36.518957 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.518920    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:57:36.519435 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.519117    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" podUID="433100f4-fe18-4234-8e37-906dc914cd95" containerName="payload-processing" containerID="cri-o://21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e" gracePeriod=30
Sep 11 12:57:36.530278 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.530246    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:57:36.530750 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.530719    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" podUID="3448dcf6-d924-4a92-8861-614c72d6b31e" containerName="payload-pre-processing" containerID="cri-o://23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e" gracePeriod=30
Sep 11 12:57:36.796726 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.796706    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:57:36.800266 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.800248    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:57:36.890232 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890151    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlgdz\" (UniqueName: \"kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz\") pod \"3448dcf6-d924-4a92-8861-614c72d6b31e\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") "
Sep 11 12:57:36.890232 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890202    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srgdk\" (UniqueName: \"kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk\") pod \"433100f4-fe18-4234-8e37-906dc914cd95\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") "
Sep 11 12:57:36.890450 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890255    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume\") pod \"433100f4-fe18-4234-8e37-906dc914cd95\" (UID: \"433100f4-fe18-4234-8e37-906dc914cd95\") "
Sep 11 12:57:36.890450 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890286    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume\") pod \"3448dcf6-d924-4a92-8861-614c72d6b31e\" (UID: \"3448dcf6-d924-4a92-8861-614c72d6b31e\") "
Sep 11 12:57:36.890673 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890650    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3448dcf6-d924-4a92-8861-614c72d6b31e" (UID: "3448dcf6-d924-4a92-8861-614c72d6b31e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:57:36.890740 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.890662    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "433100f4-fe18-4234-8e37-906dc914cd95" (UID: "433100f4-fe18-4234-8e37-906dc914cd95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:57:36.892257 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.892237    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz" (OuterVolumeSpecName: "kube-api-access-vlgdz") pod "3448dcf6-d924-4a92-8861-614c72d6b31e" (UID: "3448dcf6-d924-4a92-8861-614c72d6b31e"). InnerVolumeSpecName "kube-api-access-vlgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:57:36.892318 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.892283    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk" (OuterVolumeSpecName: "kube-api-access-srgdk") pod "433100f4-fe18-4234-8e37-906dc914cd95" (UID: "433100f4-fe18-4234-8e37-906dc914cd95"). InnerVolumeSpecName "kube-api-access-srgdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:57:36.991467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.991439    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3448dcf6-d924-4a92-8861-614c72d6b31e-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:36.991467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.991461    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgdz\" (UniqueName: \"kubernetes.io/projected/3448dcf6-d924-4a92-8861-614c72d6b31e-kube-api-access-vlgdz\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:36.991467 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.991470    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srgdk\" (UniqueName: \"kubernetes.io/projected/433100f4-fe18-4234-8e37-906dc914cd95-kube-api-access-srgdk\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:36.991674 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:36.991478    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433100f4-fe18-4234-8e37-906dc914cd95-plugins-config-volume\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 12:57:37.225585 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.225550    2575 generic.go:356] "Generic (PLEG): container finished" podID="433100f4-fe18-4234-8e37-906dc914cd95" containerID="21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e" exitCode=0
Sep 11 12:57:37.225760 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.225596    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"
Sep 11 12:57:37.225760 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.225634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" event={"ID":"433100f4-fe18-4234-8e37-906dc914cd95","Type":"ContainerDied","Data":"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"}
Sep 11 12:57:37.225760 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.225699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29" event={"ID":"433100f4-fe18-4234-8e37-906dc914cd95","Type":"ContainerDied","Data":"14a3068881f6dc33a9f9ee810aa6fad8c141e380a0877ff4b53d58c58235bfb5"}
Sep 11 12:57:37.225760 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.225735    2575 scope.go:117] "RemoveContainer" containerID="21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"
Sep 11 12:57:37.233987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.233871    2575 generic.go:356] "Generic (PLEG): container finished" podID="3448dcf6-d924-4a92-8861-614c72d6b31e" containerID="23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e" exitCode=0
Sep 11 12:57:37.233987 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.233943    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"
Sep 11 12:57:37.234122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.233964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" event={"ID":"3448dcf6-d924-4a92-8861-614c72d6b31e","Type":"ContainerDied","Data":"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"}
Sep 11 12:57:37.234122 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.234069    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j" event={"ID":"3448dcf6-d924-4a92-8861-614c72d6b31e","Type":"ContainerDied","Data":"ae44d55826a7c4d2cc6c0c2f0557e55bc5ea24396b724fcaf2f299a11f855c1d"}
Sep 11 12:57:37.241802 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.241787    2575 scope.go:117] "RemoveContainer" containerID="21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"
Sep 11 12:57:37.242091 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:57:37.242060    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e\": container with ID starting with 21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e not found: ID does not exist" containerID="21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"
Sep 11 12:57:37.242197 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.242091    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e"} err="failed to get container status \"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e\": rpc error: code = NotFound desc = could not find container \"21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e\": container with ID starting with 21dfc887956ab0e8ba0ac573ab019e086ecdf22c4b21ee64bec57cb6bc6dc39e not found: ID does not exist"
Sep 11 12:57:37.242197 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.242107    2575 scope.go:117] "RemoveContainer" containerID="23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"
Sep 11 12:57:37.249721 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.249704    2575 scope.go:117] "RemoveContainer" containerID="23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"
Sep 11 12:57:37.249990 ip-10-0-140-247 kubenswrapper[2575]: E0911 12:57:37.249964    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e\": container with ID starting with 23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e not found: ID does not exist" containerID="23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"
Sep 11 12:57:37.250040 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.249995    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e"} err="failed to get container status \"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e\": rpc error: code = NotFound desc = could not find container \"23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e\": container with ID starting with 23ab00bd5e92c3599abdeac4e12c4c7ba04d9e5ff5b31a1089c5c484a04b646e not found: ID does not exist"
Sep 11 12:57:37.256028 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.256004    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:57:37.259497 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.259476    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1337fa-fb6f7bc7f-pvx29"]
Sep 11 12:57:37.273218 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.273199    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:57:37.277849 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.277831    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1337fa-7fc994b478-5qf5j"]
Sep 11 12:57:37.629728 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.629695    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3448dcf6-d924-4a92-8861-614c72d6b31e" path="/var/lib/kubelet/pods/3448dcf6-d924-4a92-8861-614c72d6b31e/volumes"
Sep 11 12:57:37.630080 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:37.630066    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433100f4-fe18-4234-8e37-906dc914cd95" path="/var/lib/kubelet/pods/433100f4-fe18-4234-8e37-906dc914cd95/volumes"
Sep 11 12:57:57.888742 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:57:57.888711    2575 scope.go:117] "RemoveContainer" containerID="03b06d5f55bfea94de89bc9548e9317679f950e790dd6f701fba83362045ca55"
Sep 11 12:58:57.928567 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:58:57.928533    2575 scope.go:117] "RemoveContainer" containerID="db39fef33b23ba2dec1a3fc8a9ea66e8f9af6408717b08eeba1740509d102be1"
Sep 11 12:58:57.936762 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:58:57.936737    2575 scope.go:117] "RemoveContainer" containerID="4ae87ec1438c7fc300a638b4a3f8d17e4ad741c43a5f4d3fbae7f24686e85f9d"
Sep 11 12:58:57.943627 ip-10-0-140-247 kubenswrapper[2575]: I0911 12:58:57.943606    2575 scope.go:117] "RemoveContainer" containerID="6d0bcf9a6fe39511afedd246a6379b351632bda930d1bcea348783f23d7bc221"
Sep 11 13:01:57.989043 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:01:57.989013    2575 scope.go:117] "RemoveContainer" containerID="2c709a0c2194d0afd8f61052e437fecaedab30eb86c0e476b115879f75c13a7e"
Sep 11 13:01:57.997388 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:01:57.997368    2575 scope.go:117] "RemoveContainer" containerID="be3b7e4e7c7884258e561c92acf431a88851221cf41ab649dc24fd053d88e496"
Sep 11 13:02:13.579490 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.579446    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:13.580047 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580028    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee1da40-14f1-4405-b691-42dcbb7c5552" containerName="payload-processing"
Sep 11 13:02:13.580097 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580052    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee1da40-14f1-4405-b691-42dcbb7c5552" containerName="payload-processing"
Sep 11 13:02:13.580097 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580068    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433100f4-fe18-4234-8e37-906dc914cd95" containerName="payload-processing"
Sep 11 13:02:13.580097 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580078    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="433100f4-fe18-4234-8e37-906dc914cd95" containerName="payload-processing"
Sep 11 13:02:13.580097 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580096    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25f907e-11b9-447d-8703-0ac2a8c42778" containerName="payload-pre-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580105    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25f907e-11b9-447d-8703-0ac2a8c42778" containerName="payload-pre-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580128    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3448dcf6-d924-4a92-8861-614c72d6b31e" containerName="payload-pre-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580136    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3448dcf6-d924-4a92-8861-614c72d6b31e" containerName="payload-pre-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580148    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" containerName="payload-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580157    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" containerName="payload-processing"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580176    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" containerName="revoke-keys"
Sep 11 13:02:13.580225 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580185    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" containerName="revoke-keys"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580280    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee1da40-14f1-4405-b691-42dcbb7c5552" containerName="payload-processing"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580291    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68abb0f-6ccd-4dbb-a748-79ae0f3d428c" containerName="payload-processing"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580301    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25f907e-11b9-447d-8703-0ac2a8c42778" containerName="payload-pre-processing"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580311    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3448dcf6-d924-4a92-8861-614c72d6b31e" containerName="payload-pre-processing"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580338    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9f448d-5d4b-452e-8e09-8ebb4a2e9214" containerName="revoke-keys"
Sep 11 13:02:13.580474 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.580351    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="433100f4-fe18-4234-8e37-906dc914cd95" containerName="payload-processing"
Sep 11 13:02:13.583530 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.583512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.586344 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.586306    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-de400b32-kserve-self-signed-certs\""
Sep 11 13:02:13.600549 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.600525    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:13.633635 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.633740 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.633740 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.633740 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.633867 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.633867 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.633802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27hw\" (UniqueName: \"kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734573 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734712 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27hw\" (UniqueName: \"kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734712 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734712 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734712 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.734712 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.734708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.735084 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.735062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.735163 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.735136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.735163 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.735152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.736927 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.736910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.737193 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.737174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.746338 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.746310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27hw\" (UniqueName: \"kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw\") pod \"e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") " pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:13.893451 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:13.893391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:14.162442 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:14.162416    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:14.164128 ip-10-0-140-247 kubenswrapper[2575]: W0911 13:02:14.164099    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fbb8f7d_1c01_46fe_ac78_f4aa8ec2bb7c.slice/crio-f4e7a87f23bdf98e373f116bf44ca633a2a90f2167fc2c885041eba86f183f1c WatchSource:0}: Error finding container f4e7a87f23bdf98e373f116bf44ca633a2a90f2167fc2c885041eba86f183f1c: Status 404 returned error can't find the container with id f4e7a87f23bdf98e373f116bf44ca633a2a90f2167fc2c885041eba86f183f1c
Sep 11 13:02:14.513392 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:14.513358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" event={"ID":"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c","Type":"ContainerStarted","Data":"a92caa8720e665cc0e18516eaa9f3453358ea42b55acae93631c717055e37c30"}
Sep 11 13:02:14.513566 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:14.513408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" event={"ID":"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c","Type":"ContainerStarted","Data":"f4e7a87f23bdf98e373f116bf44ca633a2a90f2167fc2c885041eba86f183f1c"}
Sep 11 13:02:14.534849 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:14.534800    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" podStartSLOduration=1.53478563 podStartE2EDuration="1.53478563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:02:14.532902683 +0000 UTC m=+2117.374653833" watchObservedRunningTime="2026-09-11 13:02:14.53478563 +0000 UTC m=+2117.376536756"
Sep 11 13:02:23.894356 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:23.894304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:23.894920 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:23.894371    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:23.906141 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:23.906118    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:24.569023 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:24.568996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:41.565803 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:41.565762    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:56.584363 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.584266    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" podUID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" containerName="main" containerID="cri-o://a92caa8720e665cc0e18516eaa9f3453358ea42b55acae93631c717055e37c30" gracePeriod=45
Sep 11 13:02:56.704561 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.704534    2575 generic.go:356] "Generic (PLEG): container finished" podID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" containerID="a92caa8720e665cc0e18516eaa9f3453358ea42b55acae93631c717055e37c30" exitCode=0
Sep 11 13:02:56.704680 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.704597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" event={"ID":"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c","Type":"ContainerDied","Data":"a92caa8720e665cc0e18516eaa9f3453358ea42b55acae93631c717055e37c30"}
Sep 11 13:02:56.826684 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.826658    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:56.894765 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894690    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.894765 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894751    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q27hw\" (UniqueName: \"kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.894970 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894784    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.894970 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894807    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.894970 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894854    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.894970 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.894874    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs\") pod \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\" (UID: \"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c\") "
Sep 11 13:02:56.895174 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.895086    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:02:56.895174 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.895111    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:02:56.895262 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.895199    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home" (OuterVolumeSpecName: "home") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:02:56.895262 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.895217    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-model-cache\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:56.895262 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.895239    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tmp-dir\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:56.896842 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.896819    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw" (OuterVolumeSpecName: "kube-api-access-q27hw") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "kube-api-access-q27hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:02:56.896946 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.896820    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm" (OuterVolumeSpecName: "dshm") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:02:56.896946 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.896845    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" (UID: "9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:02:56.996607 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.996566    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-dshm\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:56.996607 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.996602    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27hw\" (UniqueName: \"kubernetes.io/projected/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-kube-api-access-q27hw\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:56.996607 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.996613    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-home\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:56.996821 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:56.996622    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c-tls-certs\") on node \"ip-10-0-140-247.ec2.internal\" DevicePath \"\""
Sep 11 13:02:57.710854 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:57.710826    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"
Sep 11 13:02:57.711249 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:57.710829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48" event={"ID":"9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c","Type":"ContainerDied","Data":"f4e7a87f23bdf98e373f116bf44ca633a2a90f2167fc2c885041eba86f183f1c"}
Sep 11 13:02:57.711249 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:57.710951    2575 scope.go:117] "RemoveContainer" containerID="a92caa8720e665cc0e18516eaa9f3453358ea42b55acae93631c717055e37c30"
Sep 11 13:02:57.728374 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:57.728350    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:57.733108 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:57.733087    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-de400b32-kserve-84585ffcbc-b2h48"]
Sep 11 13:02:58.017284 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:58.017253    2575 scope.go:117] "RemoveContainer" containerID="adc246cdb6c27b775167f3f718556af604bfcc1867dfe1c7de90c83d227cd76e"
Sep 11 13:02:59.629109 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:02:59.629074    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" path="/var/lib/kubelet/pods/9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c/volumes"
Sep 11 13:03:32.862454 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:32.862419    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pjgsm_b8244cc6-7365-4a81-b719-eea621e569c1/manager/0.log"
Sep 11 13:03:33.198579 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:33.198553    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65cbc8d955-g4s6r_1c45116d-1144-4aac-b3aa-a2222db72246/copy-manifests/0.log"
Sep 11 13:03:33.217201 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:33.217176    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65cbc8d955-g4s6r_1c45116d-1144-4aac-b3aa-a2222db72246/manager/0.log"
Sep 11 13:03:35.756697 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:35.756666    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h_a5355b5d-088b-4b2b-ad39-02b8492e1088/istio-proxy/0.log"
Sep 11 13:03:35.987502 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:35.987469    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67c8b95cc4-hzmfv_4a7eb3a4-028e-424d-a26f-58a29f2e5826/kube-auth-proxy/0.log"
Sep 11 13:03:36.459671 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:36.459646    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 13:03:37.121001 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:37.120972    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vqsgm_2ef48ca8-dc30-4471-a565-7100af08435f/main/0.log"
Sep 11 13:03:37.227887 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:37.227856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xqhf4_8591664a-e618-4d86-994c-3292349cbc98/main/0.log"
Sep 11 13:03:37.333024 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:37.333002    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flzf4l_131fa2c8-be72-48d7-9563-34275474b9d3/main/0.log"
Sep 11 13:03:37.561908 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:37.561877    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-m8bnx_c8d9639b-3bac-4804-8475-ef092fb1f18f/main/0.log"
Sep 11 13:03:45.669256 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:45.669206    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mzvww_74dbc65a-330f-406e-9f26-34cd5e77565d/global-pull-secret-syncer/0.log"
Sep 11 13:03:45.809763 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:45.809721    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tx7bj_ebfa985d-6c0e-4bea-be94-078f3bef8d04/konnectivity-agent/0.log"
Sep 11 13:03:45.861317 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:45.861290    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-247.ec2.internal_8e6f1705d022cd6d1f69ce3536d23a7e/haproxy/0.log"
Sep 11 13:03:51.631255 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:51.631223    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d75b47df-5q5bq_bb39a362-71b7-4626-9b88-0adc8f9544af/metrics-server/0.log"
Sep 11 13:03:51.657640 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:51.657610    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-d955z_5895766c-ffbb-4967-bc4d-13cc14a1c6db/monitoring-plugin/0.log"
Sep 11 13:03:51.769019 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:51.768990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/node-exporter/0.log"
Sep 11 13:03:51.792076 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:51.792049    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/kube-rbac-proxy/0.log"
Sep 11 13:03:51.815880 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:51.815861    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd4lz_d37e9227-9205-48a9-85ed-f0fd03c0d37b/init-textfile/0.log"
Sep 11 13:03:52.363606 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.363575    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/thanos-query/0.log"
Sep 11 13:03:52.383140 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.383113    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-web/0.log"
Sep 11 13:03:52.401488 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.401464    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy/0.log"
Sep 11 13:03:52.424508 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.424486    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/prom-label-proxy/0.log"
Sep 11 13:03:52.445901 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.445880    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-rules/0.log"
Sep 11 13:03:52.467898 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:52.467874    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79bbf8d7d5-xmm5m_909cb820-6630-4062-9827-a8c3009c0642/kube-rbac-proxy-metrics/0.log"
Sep 11 13:03:55.387953 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.387914    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"]
Sep 11 13:03:55.388447 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.388427    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" containerName="main"
Sep 11 13:03:55.388530 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.388450    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" containerName="main"
Sep 11 13:03:55.388584 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.388560    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbb8f7d-1c01-46fe-ac78-f4aa8ec2bb7c" containerName="main"
Sep 11 13:03:55.391194 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.391173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.394371 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.394349    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rhplr\"/\"default-dockercfg-lbvs9\""
Sep 11 13:03:55.394473 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.394368    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhplr\"/\"openshift-service-ca.crt\""
Sep 11 13:03:55.395676 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.395659    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhplr\"/\"kube-root-ca.crt\""
Sep 11 13:03:55.401204 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.401184    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"]
Sep 11 13:03:55.489038 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.489011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-proc\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.489135 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.489043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-lib-modules\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.489135 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.489064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-sys\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.489211 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.489143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-podres\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.489211 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.489182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdgq\" (UniqueName: \"kubernetes.io/projected/07f80ab3-4af1-48cf-b40d-8bc37172194e-kube-api-access-xbdgq\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590241 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-proc\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590318 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-lib-modules\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590318 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-sys\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590425 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-podres\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590425 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590359    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-proc\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590425 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdgq\" (UniqueName: \"kubernetes.io/projected/07f80ab3-4af1-48cf-b40d-8bc37172194e-kube-api-access-xbdgq\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590425 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-lib-modules\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590574 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-sys\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.590574 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.590466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07f80ab3-4af1-48cf-b40d-8bc37172194e-podres\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.601238 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.601213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdgq\" (UniqueName: \"kubernetes.io/projected/07f80ab3-4af1-48cf-b40d-8bc37172194e-kube-api-access-xbdgq\") pod \"perf-node-gather-daemonset-4dbkz\" (UID: \"07f80ab3-4af1-48cf-b40d-8bc37172194e\") " pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.701218 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.701199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:55.970234 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:55.970139    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"]
Sep 11 13:03:55.972565 ip-10-0-140-247 kubenswrapper[2575]: W0911 13:03:55.972540    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07f80ab3_4af1_48cf_b40d_8bc37172194e.slice/crio-bdec6537f0b5b617b4ab9188029b341e81174ddc429263514af8644b8132e4b4 WatchSource:0}: Error finding container bdec6537f0b5b617b4ab9188029b341e81174ddc429263514af8644b8132e4b4: Status 404 returned error can't find the container with id bdec6537f0b5b617b4ab9188029b341e81174ddc429263514af8644b8132e4b4
Sep 11 13:03:56.977891 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:56.977848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz" event={"ID":"07f80ab3-4af1-48cf-b40d-8bc37172194e","Type":"ContainerStarted","Data":"ce2df83e7014303c46e55375105d3c11e8c7a29c4047037220b5ae3381cc4fd1"}
Sep 11 13:03:56.978302 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:56.977912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz" event={"ID":"07f80ab3-4af1-48cf-b40d-8bc37172194e","Type":"ContainerStarted","Data":"bdec6537f0b5b617b4ab9188029b341e81174ddc429263514af8644b8132e4b4"}
Sep 11 13:03:56.978302 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:56.977954    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:03:57.004319 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.004261    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz" podStartSLOduration=2.004246375 podStartE2EDuration="2.004246375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:03:57.003640676 +0000 UTC m=+2219.845391823" watchObservedRunningTime="2026-09-11 13:03:57.004246375 +0000 UTC m=+2219.845997500"
Sep 11 13:03:57.271155 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.271071    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wsqbf_e364c8eb-842d-4bf4-98b2-5db29632f4af/dns/0.log"
Sep 11 13:03:57.296429 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.296394    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wsqbf_e364c8eb-842d-4bf4-98b2-5db29632f4af/kube-rbac-proxy/0.log"
Sep 11 13:03:57.322924 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.322892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5sw85_de85618e-7d30-44b9-87f0-dda082e4718e/dns-node-resolver/0.log"
Sep 11 13:03:57.901110 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.901071    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-574c6bf895-twjkf_6e4dcdc3-844b-4c4b-a07f-0d26be59bf6a/registry/0.log"
Sep 11 13:03:57.966702 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:57.966676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t48d7_8dd90914-8e03-4ea0-bfe8-bd7a31341a37/node-ca/0.log"
Sep 11 13:03:58.853514 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:58.853482    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h_a5355b5d-088b-4b2b-ad39-02b8492e1088/istio-proxy/0.log"
Sep 11 13:03:59.125801 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:59.125727    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67c8b95cc4-hzmfv_4a7eb3a4-028e-424d-a26f-58a29f2e5826/kube-auth-proxy/0.log"
Sep 11 13:03:59.293049 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:03:59.293022    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 13:04:00.196832 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:00.196792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnsxs_ffcfa61d-a48d-4a8f-b176-56b9fe2d1d12/serve-healthcheck-canary/0.log"
Sep 11 13:04:00.807955 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:00.807925    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdxzw_56774c0e-65c3-4ca1-ac11-83db1757dc0d/kube-rbac-proxy/0.log"
Sep 11 13:04:00.827799 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:00.827779    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdxzw_56774c0e-65c3-4ca1-ac11-83db1757dc0d/exporter/0.log"
Sep 11 13:04:00.849401 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:00.849363    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdxzw_56774c0e-65c3-4ca1-ac11-83db1757dc0d/extractor/0.log"
Sep 11 13:04:02.991567 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:02.991540    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rhplr/perf-node-gather-daemonset-4dbkz"
Sep 11 13:04:03.383352 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:03.383255    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pjgsm_b8244cc6-7365-4a81-b719-eea621e569c1/manager/0.log"
Sep 11 13:04:03.606751 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:03.606715    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65cbc8d955-g4s6r_1c45116d-1144-4aac-b3aa-a2222db72246/manager/0.log"
Sep 11 13:04:03.625713 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:03.625686    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65cbc8d955-g4s6r_1c45116d-1144-4aac-b3aa-a2222db72246/copy-manifests/0.log"
Sep 11 13:04:04.974085 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:04.974058    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-556b8bb6b-r6tn2_44b835db-4097-4cf0-8840-b3f4744a59bf/manager/0.log"
Sep 11 13:04:05.048564 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:05.048506    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7bvf9_f84210d2-f492-4ccf-9706-222a31f10f2d/openshift-lws-operator/0.log"
Sep 11 13:04:05.101399 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:05.101368    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h_a5355b5d-088b-4b2b-ad39-02b8492e1088/istio-proxy/0.log"
Sep 11 13:04:05.324281 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:05.324215    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67c8b95cc4-hzmfv_4a7eb3a4-028e-424d-a26f-58a29f2e5826/kube-auth-proxy/0.log"
Sep 11 13:04:05.432253 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:05.432227    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 13:04:06.834863 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:06.834835    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnp5h_a5355b5d-088b-4b2b-ad39-02b8492e1088/istio-proxy/0.log"
Sep 11 13:04:07.077617 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:07.077580    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67c8b95cc4-hzmfv_4a7eb3a4-028e-424d-a26f-58a29f2e5826/kube-auth-proxy/0.log"
Sep 11 13:04:07.187381 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:07.187350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zzfmq_d2e3eecf-bfdc-4886-a890-ba8f58f5a8b6/payload-processing/0.log"
Sep 11 13:04:09.903503 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:09.903473    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-n47v2_e9efa306-b755-430c-a151-1161d9292d43/migrator/0.log"
Sep 11 13:04:09.923164 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:09.923143    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-n47v2_e9efa306-b755-430c-a151-1161d9292d43/graceful-termination/0.log"
Sep 11 13:04:11.376041 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.376008    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9hjxt_61a373b3-a7f6-4108-bd1d-604a51547fb3/kube-multus/0.log"
Sep 11 13:04:11.740932 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.740908    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/kube-multus-additional-cni-plugins/0.log"
Sep 11 13:04:11.780123 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.780102    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/egress-router-binary-copy/0.log"
Sep 11 13:04:11.819926 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.819905    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/cni-plugins/0.log"
Sep 11 13:04:11.854917 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.854895    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/bond-cni-plugin/0.log"
Sep 11 13:04:11.887828 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.887805    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/routeoverride-cni/0.log"
Sep 11 13:04:11.920865 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.920846    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/whereabouts-cni-bincopy/0.log"
Sep 11 13:04:11.961785 ip-10-0-140-247 kubenswrapper[2575]: I0911 13:04:11.961763    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9b6k9_50f30488-bba7-4cb1-83ec-e9e58eb6000e/whereabouts-cni/0.log"