Sep 09 12:11:17.249605 ip-10-0-128-117 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 12:11:17.673045 ip-10-0-128-117 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:11:17.673045 ip-10-0-128-117 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 12:11:17.673045 ip-10-0-128-117 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:11:17.673045 ip-10-0-128-117 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 12:11:17.673045 ip-10-0-128-117 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 12:11:17.674795 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.674702    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679350    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679374    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679378    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679381    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679383    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679386    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.679378 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679389    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679392    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679395    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679397    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679400    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679403    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679405    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679408    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679410    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679412    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679415    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679417    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679420    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679422    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679425    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679427    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679429    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679432    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679434    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.679621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679437    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679442    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679445    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679447    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679450    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679452    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679455    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679457    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679460    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679462    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679465    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679467    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679470    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679472    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679474    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679478    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679480    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679484    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679486    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679489    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.680097 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679491    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679494    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679496    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679499    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679501    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679504    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679506    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679509    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679512    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679514    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679516    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679519    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679521    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679524    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679526    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679529    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679531    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679534    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679536    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.680724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679538    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679543    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679547    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679550    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679553    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679556    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679559    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679561    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679564    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679568    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679570    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679573    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679577    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679581    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679585    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679588    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679591    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679594    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679596    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.681229 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679600    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679603    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679606    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679609    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679611    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679614    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679617    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679619    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679622    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679625    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679628    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679630    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679633    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679636    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679638    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.679641    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681152    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681160    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681164    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681168    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.681711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681171    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681175    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681178    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681181    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681184    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681186    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681189    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681192    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681194    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681197    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681199    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681202    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681204    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681208    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681211    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681213    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681216    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681218    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681221    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681223    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.682201 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681226    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681228    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681231    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681233    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681252    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681255    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681257    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681260    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681263    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681265    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681268    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681271    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681273    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681276    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681279    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681281    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681285    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681290    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681293    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681297    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.682732 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681299    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681302    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681304    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681307    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681310    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681312    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681315    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681318    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681320    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681322    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681325    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681327    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681330    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681332    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681335    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681338    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681341    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681343    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681348    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.683228 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681352    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681354    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681357    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681359    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681362    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681365    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681368    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681370    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681373    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681375    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681378    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681380    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681383    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681385    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681388    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681390    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681393    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681395    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681397    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681400    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.683709 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681404    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681406    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681409    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681411    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681414    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681416    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681419    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681421    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681424    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681426    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681429    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681432    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681436    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681439    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681442    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681445    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681819    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681824    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681827    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681830    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.684207 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681832    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681835    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681837    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681840    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681842    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681845    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681847    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681850    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681852    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681855    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681857    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681860    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681863    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681866    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681871    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681875    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681877    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681880    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681883    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.684724 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681886    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681888    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681891    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681894    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681896    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681899    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681902    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681904    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681907    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681909    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681912    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681916    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681919    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681921    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681924    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681926    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681928    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681931    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681933    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681936    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.685193 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681938    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681941    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681943    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681946    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681948    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681951    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681954    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681956    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681959    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681962    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681964    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681967    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681969    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681971    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681974    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681976    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681979    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681981    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681984    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681986    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.685740 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681989    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681991    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681994    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681997    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.681999    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682002    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682004    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682007    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682009    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682011    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682014    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682016    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682019    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682022    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682025    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682028    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682030    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682033    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682036    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.686296 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682043    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682047    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682050    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682053    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682056    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682058    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682061    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682064    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682067    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682070    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682072    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682074    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682077    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682079    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682081    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682084    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682086    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682158    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682201    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682208    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682213    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 12:11:17.686767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682217    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682220    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682225    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682229    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682232    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682248    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682252    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682255    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682258    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682261    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682264    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682267    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682271    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682273    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682277    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682280    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682283    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682286    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682289    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682293    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682296    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682298    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682301    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682304    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 12:11:17.687285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682307    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682310    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682313    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682315    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682319    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682322    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682325    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682328    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682331    2556 flags.go:64] FLAG: --enable-server="true"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682333    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682339    2556 flags.go:64] FLAG: --event-burst="100"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682342    2556 flags.go:64] FLAG: --event-qps="50"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682345    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682348    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682351    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682355    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682357    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682360    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682363    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682366    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682369    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682372    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682375    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682378    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682380    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 12:11:17.687868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682383    2556 flags.go:64] FLAG: --feature-gates=""
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682387    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682390    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682393    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682396    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682399    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682401    2556 flags.go:64] FLAG: --help="false"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682404    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-128-117.ec2.internal"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682407    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682410    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682413    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682416    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682419    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682422    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682425    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682428    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682430    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682435    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682438    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682443    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682445    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682448    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682451    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682453    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 12:11:17.688529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682456    2556 flags.go:64] FLAG: --lock-file=""
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682459    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682462    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682465    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682470    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682474    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682476    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682479    2556 flags.go:64] FLAG: --logging-format="text"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682482    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682485    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682487    2556 flags.go:64] FLAG: --manifest-url=""
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682490    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682495    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682497    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682501    2556 flags.go:64] FLAG: --max-pods="110"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682504    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682507    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682510    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682513    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682515    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682518    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682521    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682528    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682531    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682534    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 12:11:17.689155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682538    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682541    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682545    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682550    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682553    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682555    2556 flags.go:64] FLAG: --port="10250"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682558    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682561    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089c0f387c5aeb8d6"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682564    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682568    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682570    2556 flags.go:64] FLAG: --register-node="true"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682573    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682577    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682580    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682583    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682585    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682589    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682592    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682595    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682598    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682600    2556 flags.go:64] FLAG: --runonce="false"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682603    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682606    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682608    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682611    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 12:11:17.689824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682614    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682617    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682620    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682623    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682625    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682628    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682631    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682634    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682637    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682640    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682645    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682651    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682654    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682656    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682660    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682663    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682666    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682669    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682672    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682675    2556 flags.go:64] FLAG: --v="2"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682678    2556 flags.go:64] FLAG: --version="false"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682683    2556 flags.go:64] FLAG: --vmodule=""
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682686    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.682690    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682803    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.690463 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682807    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682809    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682812    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682815    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682817    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682820    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682823    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682826    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682828    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682831    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682834    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682837    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682839    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682842    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682845    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682848    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682851    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682855    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682858    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.691070 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682860    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682863    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682866    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682868    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682871    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682874    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682876    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682879    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682881    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682884    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682887    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682889    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682892    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682895    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682897    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682899    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682902    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682904    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682907    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682909    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.691565 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682912    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682914    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682917    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682919    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682922    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682924    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682927    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682930    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682933    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682936    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682939    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682942    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682944    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682947    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682951    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682955    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682957    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682961    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682964    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682966    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.692063 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682969    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682971    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682974    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682979    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682984    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682988    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682992    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682996    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.682999    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683003    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683007    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683009    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683012    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683015    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683018    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683020    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683023    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683025    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683028    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.692681 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683031    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683034    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683036    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683040    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683042    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683045    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683047    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683050    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683052    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683055    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683058    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683060    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683063    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683065    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683067    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683070    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683072    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683076    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683078    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.693139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.683081    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.683686    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691094    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691108    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691127    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691133    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691188    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691193    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691196    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691200    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691203    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691206    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691209    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691213    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691217    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.693613 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691220    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691223    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691226    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691228    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691232    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691252    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691257    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691260    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691263    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691265    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691268    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691271    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691274    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691277    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691279    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691283    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691285    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691288    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691291    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691294    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.693994 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691297    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691300    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691302    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691305    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691307    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691310    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691313    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691315    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691318    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691320    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691322    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691325    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691328    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691330    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691333    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691335    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691338    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691340    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691343    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691345    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.694510 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691348    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691350    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691352    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691355    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691357    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691360    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691362    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691364    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691367    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691370    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691372    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691374    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691378    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691380    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691383    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691385    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691388    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691390    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691393    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691395    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.695001 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691398    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691400    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691402    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691405    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691407    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691410    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691414    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691417    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691419    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691422    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691424    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691427    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691429    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691432    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691434    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691437    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691439    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691442    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691444    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.695519 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691447    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691450    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691452    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691455    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691457    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691461    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691463    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691466    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691468    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691472    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691475    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691480    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691589    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691594    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 12:11:17.695989 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691597    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691600    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691603    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691606    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691608    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691611    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691614    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691617    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691620    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691622    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691626    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691630    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691632    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691635    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691638    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691640    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691643    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691645    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691648    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 12:11:17.696442 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691650    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691652    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691659    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691661    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691665    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691669    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691672    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691675    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691677    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691680    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691683    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691686    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691688    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691691    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691693    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691696    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691698    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691700    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691703    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691705    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 12:11:17.696898 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691708    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691710    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691713    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691715    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691718    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691721    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691724    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691727    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691730    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691732    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691735    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691738    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691740    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691743    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691745    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691748    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691751    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691754    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691756    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 12:11:17.697401 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691759    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691761    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691764    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691766    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691769    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691771    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691774    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691776    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691778    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691781    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691783    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691786    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691788    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691791    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691793    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691795    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691798    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691800    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691803    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691805    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 12:11:17.697924 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691808    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691810    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691812    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691815    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691817    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691819    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691822    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691824    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691827    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691830    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691832    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691834    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691837    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691839    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691842    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691844    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691846    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691849    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 12:11:17.698465 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:17.691852    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691856    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.691968    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.694018    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.695029    2556 server.go:1019] "Starting client certificate rotation"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.695124    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.695170    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.697527    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:11:17.698958 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.698836    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 12:11:17.707536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.707515    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 09 12:11:17.711325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.711307    2556 log.go:25] "Validated CRI v1 image API"
Sep 09 12:11:17.712203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.712186    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 12:11:17.717715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.717682    2556 fs.go:135] Filesystem UUIDs: map[3cf804db-ca9a-40fc-9f42-921da9381bfb:/dev/nvme0n1p4 4fd25116-ef2d-45fe-a828-31d1b88ffcd4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 12:11:17.717796 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.717712    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 12:11:17.724387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.724268    2556 manager.go:216] Machine: {Timestamp:2026-09-09 12:11:17.721411937 +0000 UTC m=+0.351999715 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3033276 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f61ca9b2847d73b8cbad57af0a41d SystemUUID:ec2f61ca-9b28-47d7-3b8c-bad57af0a41d BootID:f728710f-c99a-4433-836f-32ce016eb0f7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:8e:e6:79:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:8e:e6:79:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:82:99:55:30:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 12:11:17.724387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.724378    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 12:11:17.724550 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.724490    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 12:11:17.724994 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.724975    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:11:17.728717 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.728689    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 12:11:17.728883 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.728722    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 12:11:17.730197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.730185    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 12:11:17.730281 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.730201    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 12:11:17.730281 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.730212    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:11:17.730912 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.730899    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 12:11:17.732251 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.732228    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 12:11:17.735752 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735738    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:11:17.735878 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735867    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 12:11:17.735955 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735913    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 12:11:17.735955 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735928    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 12:11:17.735955 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735951    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 12:11:17.736075 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735966    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 09 12:11:17.736075 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.735980    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 12:11:17.736380 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.736366    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:11:17.736439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.736390    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 12:11:17.738743 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.738723    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 12:11:17.741498 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741484    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 12:11:17.741563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741515    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 12:11:17.741669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741660    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 12:11:17.741703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741674    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 12:11:17.741703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741680    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 12:11:17.741703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741686    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 12:11:17.741703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741694    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 12:11:17.741703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.741699    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 12:11:17.742367 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742355    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 12:11:17.742413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742370    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 12:11:17.742413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742381    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 12:11:17.742413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742389    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 12:11:17.742413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742401    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 12:11:17.742413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.742413    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 12:11:17.743755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.743743    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 12:11:17.743755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.743755    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 12:11:17.744900 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.744871    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 12:11:17.744900 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.744880    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 12:11:17.748466 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.748447    2556 server.go:1268] "Started kubelet"
Sep 09 12:11:17.748559 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.748505    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 12:11:17.748615 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.748591    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 12:11:17.748661 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.748640    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 12:11:17.749416 ip-10-0-128-117 systemd[1]: Started Kubernetes Kubelet.
Sep 09 12:11:17.749965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.749941    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 12:11:17.750603 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.750431    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 12:11:17.750713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.750686    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 12:11:17.752064 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.752045    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w7gb7"
Sep 09 12:11:17.753689 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.753662    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 12:11:17.754965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.754948    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 12:11:17.755734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.755719    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 12:11:17.755858 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.755833    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:17.755985 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.754757    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-117.ec2.internal.18d3a5e47c0a6fc3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-117.ec2.internal,UID:ip-10-0-128-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-117.ec2.internal,},FirstTimestamp:2026-09-09 12:11:17.748404163 +0000 UTC m=+0.378991930,LastTimestamp:2026-09-09 12:11:17.748404163 +0000 UTC m=+0.378991930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-117.ec2.internal,}"
Sep 09 12:11:17.756407 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756392    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 12:11:17.756514 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756499    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 12:11:17.756575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756411    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 12:11:17.756575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756566    2556 factory.go:55] Registering systemd factory
Sep 09 12:11:17.756652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756579    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 09 12:11:17.756652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756606    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 12:11:17.756652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756616    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 12:11:17.756840 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756824    2556 factory.go:153] Registering CRI-O factory
Sep 09 12:11:17.756887 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756844    2556 factory.go:223] Registration of the crio container factory successfully
Sep 09 12:11:17.756925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756900    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 12:11:17.756925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756923    2556 factory.go:103] Registering Raw factory
Sep 09 12:11:17.756998 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.756940    2556 manager.go:1196] Started watching for new ooms in manager
Sep 09 12:11:17.757387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.757375    2556 manager.go:349] Starting recovery of all containers
Sep 09 12:11:17.760660 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.760635    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w7gb7"
Sep 09 12:11:17.762721 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.762690    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 12:11:17.762863 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.762842    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 12:11:17.767051 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.767023    2556 manager.go:354] Recovery completed
Sep 09 12:11:17.772393 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.772378    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.774155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774135    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:17.774231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774171    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:17.774231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774181    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:17.774789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774771    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 12:11:17.774789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774781    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.774789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774785    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 12:11:17.774951 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.774801    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 12:11:17.775532 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.775517    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:17.775581 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.775543    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:17.775581 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.775554    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:17.775999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.775988    2556 policy_none.go:49] "None policy: Start"
Sep 09 12:11:17.776030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.776004    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 12:11:17.776030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.776013    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 12:11:17.777131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.777121    2556 policy_none.go:47] "Start"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.818846    2556 manager.go:337] "Starting Device Plugin manager"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.818887    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.818901    2556 server.go:87] "Starting device plugin registration server"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.819232    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.819281    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.819385    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.819453    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.819462    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.819947    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 12:11:17.822970 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.819989    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:17.893906 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.893868    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 12:11:17.895310 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.895293    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 12:11:17.895375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.895353    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 12:11:17.895411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.895378    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 12:11:17.895448 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.895418    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 12:11:17.897718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.897698    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:17.919522 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.919498    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.921212 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.921191    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:17.921292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.921230    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:17.921292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.921262    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:17.921292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.921287    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-117.ec2.internal"
Sep 09 12:11:17.929411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.929339    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-117.ec2.internal"
Sep 09 12:11:17.929411 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.929366    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-117.ec2.internal\": node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:17.945306 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:17.945287    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:17.996414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.996383    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"]
Sep 09 12:11:17.996549 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.996465    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.997491 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.997467    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:17.997565 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.997503    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:17.997565 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.997515    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:17.998755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.998743    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.998880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.998867    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:17.998923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.998896    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:17.999503 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.999483    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:17.999614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.999510    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:17.999614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.999523    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:18.000008 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:17.999991    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:18.000072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.000027    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:18.000072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.000045    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:18.001433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.001416    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.001500 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.001457    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 12:11:18.002167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.002153    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 12:11:18.002250 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.002177    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 12:11:18.002250 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.002188    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeHasSufficientPID"
Sep 09 12:11:18.018815 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.018794    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-117.ec2.internal\" not found" node="ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.023086 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.023069    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-117.ec2.internal\" not found" node="ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.045654 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.045628    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.058145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.058127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.058231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.058159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.058231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.058177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/707ed0fa4514c61dc617937a9e19c4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-128-117.ec2.internal\" (UID: \"707ed0fa4514c61dc617937a9e19c4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.146786 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.146754    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.159070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.159124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.159124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/707ed0fa4514c61dc617937a9e19c4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-128-117.ec2.internal\" (UID: \"707ed0fa4514c61dc617937a9e19c4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.159194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159145    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/707ed0fa4514c61dc617937a9e19c4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-128-117.ec2.internal\" (UID: \"707ed0fa4514c61dc617937a9e19c4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.159194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.159194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.159153    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9542204e0b5b72ab121300d1ff50143-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal\" (UID: \"c9542204e0b5b72ab121300d1ff50143\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.247486 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.247454    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.322931 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.322907    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.327396 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.327140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:18.348537 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.348503    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.449080 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.449036    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.549652 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.549579    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.612331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.612306    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:18.626825 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.626799    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:18.649843 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.649815    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.695355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.695323    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 12:11:18.695789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.695468    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:11:18.695789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.695503    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:11:18.695789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.695506    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 12:11:18.750824 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.750787    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.762609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.762576    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 12:06:18 +0000 UTC" deadline="2028-04-30 20:49:13.9060021 +0000 UTC"
Sep 09 12:11:18.762609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.762605    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14384h37m55.143399851s"
Sep 09 12:11:18.850980 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.850910    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.868476 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.868458    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:11:18.899633 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.899584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal" event={"ID":"c9542204e0b5b72ab121300d1ff50143","Type":"ContainerStarted","Data":"b6169239199654ac034577dd05d8dcae23f5b9837b7092e019acbeb8c5fe5934"}
Sep 09 12:11:18.900698 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.900675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal" event={"ID":"707ed0fa4514c61dc617937a9e19c4ec","Type":"ContainerStarted","Data":"97a6d39891c88147f8699778fafddbc826ae3a4c2b098a7aee331170c809ff50"}
Sep 09 12:11:18.952525 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:18.952482    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-117.ec2.internal\" not found"
Sep 09 12:11:18.996726 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:18.996694    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:19.056784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.056745    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:19.071979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.071955    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:11:19.073468 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.073448    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal"
Sep 09 12:11:19.082521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.082496    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 12:11:19.613832 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.613792    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:19.737380 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.737343    2556 apiserver.go:52] "Watching apiserver"
Sep 09 12:11:19.751373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.751341    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 12:11:19.752695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.752668    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wctrv","openshift-network-diagnostics/network-check-target-5gm6c","kube-system/konnectivity-agent-qs7x4","openshift-dns/node-resolver-gqw7g","openshift-image-registry/node-ca-bm5b2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal","openshift-multus/multus-6fjrr","openshift-network-operator/iptables-alerter-kcmhz","openshift-ovn-kubernetes/ovnkube-node-snxqt","kube-system/global-pull-secret-syncer-l7hxr","kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx","openshift-cluster-node-tuning-operator/tuned-85dn7","openshift-multus/multus-additional-cni-plugins-d6xx4"]
Sep 09 12:11:19.755276 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.754570    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.755515 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.755496    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 12:11:19.758128 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.758107    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.760087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.760069    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.760624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.760603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.762289 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.761840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.763721 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.763462    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.764671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.764648    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.765980 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.765962    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.766097 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.766060    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:19.767342 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767321    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:19.767437 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.767391    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:19.767528 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767499    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-etc-kubernetes\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767595 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-hostroot\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767595 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.767702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767597    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-var-lib-kubelet\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.767702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cni-binary-copy\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-socket-dir-parent\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-conf-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-modprobe-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-bin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-multus-certs\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767828    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-kubernetes\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-daemon-config\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wcr\" (UniqueName: \"kubernetes.io/projected/13e8d794-ed88-440f-b2fe-e5f36f718e5b-kube-api-access-s4wcr\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.767953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-host\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.767968    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-system-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-multus\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-kubelet\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-tuned\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6d6g\" (UniqueName: \"kubernetes.io/projected/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-kube-api-access-z6d6g\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-netns\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysconfig\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-systemd\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768301    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-sys\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-tmp\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cnibin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768371    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-lib-modules\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768396    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-conf\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-run\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.768833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-k8s-cni-cncf-io\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.769180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.768951    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.769180 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.769003    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:19.771216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.771195    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.773283 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.773213    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.776470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.776150    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.776470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.776284    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 12:11:19.776470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.776150    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.776677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.776580    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 12:11:19.777422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.777182    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.777422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.777214    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.777422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.777214    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 12:11:19.778545 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.778520    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.778750 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.778732    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779240    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779281    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779329    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779337    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779289    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 12:11:19.779519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779242    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 12:11:19.779864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779581    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jhk46\""
Sep 09 12:11:19.779864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779636    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cndk5\""
Sep 09 12:11:19.779973 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.779943    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.780199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780043    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7gp54\""
Sep 09 12:11:19.780199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780138    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vddw5\""
Sep 09 12:11:19.780405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780387    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 12:11:19.780484 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780416    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.780548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780388    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 12:11:19.780629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.780615    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 12:11:19.787021 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787001    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 12:11:19.787200 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787183    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-76cm8\""
Sep 09 12:11:19.787273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787212    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d7w86\""
Sep 09 12:11:19.787469 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787448    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9gnl\""
Sep 09 12:11:19.787642 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787619    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 12:11:19.787741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787669    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9lb5h\""
Sep 09 12:11:19.787867 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787847    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 12:11:19.787976 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787957    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.788041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787958    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 12:11:19.788098 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.788048    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g92kb\""
Sep 09 12:11:19.788155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.787849    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 12:11:19.788439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.788418    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 12:11:19.796403 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.796317    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 12:11:19.825784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.825744    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wsq5n"
Sep 09 12:11:19.837230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.837193    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wsq5n"
Sep 09 12:11:19.858862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.858839    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 12:11:19.868859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-config\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.868859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-kubernetes\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868870    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-daemon-config\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wcr\" (UniqueName: \"kubernetes.io/projected/13e8d794-ed88-440f-b2fe-e5f36f718e5b-kube-api-access-s4wcr\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868924    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b1b1875-e67e-49de-9073-0333de341328-serviceca\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.868999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-log-socket\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-host\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-system-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-registration-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-system-cni-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-netns\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/65f66ff2-761b-4503-be4d-8bfb9e55536e-kube-api-access-wvlng\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869211    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-kubelet-config\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-multus\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-slash\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-bin\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthgj\" (UniqueName: \"kubernetes.io/projected/a4969518-540b-43c6-af49-9166938fd718-kube-api-access-nthgj\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869387    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysconfig\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-systemd\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.869436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-sys\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-tmp\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cnibin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-hosts-file\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-lib-modules\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b1b1875-e67e-49de-9073-0333de341328-host\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-conf\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-run\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869718    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-socket-dir-parent\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-systemd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-modprobe-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwhg\" (UniqueName: \"kubernetes.io/projected/8264c391-d5f3-44d2-b6cc-e00960c29712-kube-api-access-kbwhg\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869811    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4h7\" (UniqueName: \"kubernetes.io/projected/956af29a-af32-4d5a-8eaa-ce00d3b27996-kube-api-access-mr4h7\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869833    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-systemd-units\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869852    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-env-overrides\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-sys-fs\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd313698-c628-4132-8a0e-bbcc55063c11-ovn-node-metrics-cert\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-socket-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.869982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-var-lib-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870030    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-ovn\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplnk\" (UniqueName: \"kubernetes.io/projected/dd313698-c628-4132-8a0e-bbcc55063c11-kube-api-access-wplnk\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-dbus\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-kubelet\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-tuned\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6d6g\" (UniqueName: \"kubernetes.io/projected/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-kube-api-access-z6d6g\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-netns\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqf92\" (UniqueName: \"kubernetes.io/projected/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-kube-api-access-qqf92\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-etc-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.870970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870270    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-tmp-dir\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-multus\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-cnibin\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4969518-540b-43c6-af49-9166938fd718-host-slash\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870378    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-konnectivity-ca\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870394    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysconfig\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870407    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-kubelet\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-systemd\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-agent-certs\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-sys\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-k8s-cni-cncf-io\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870536    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-k8s-cni-cncf-io\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-etc-kubernetes\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870566    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-etc-kubernetes\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870578    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.871567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870629    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4969518-540b-43c6-af49-9166938fd718-iptables-alerter-script\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-hostroot\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-kubernetes\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-var-lib-kubelet\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cni-binary-copy\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870765    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-conf-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870789    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-node-log\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870813    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-script-lib\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870864    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-bin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-multus-certs\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qqv\" (UniqueName: \"kubernetes.io/projected/5b1b1875-e67e-49de-9073-0333de341328-kube-api-access-z6qqv\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-device-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871053    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-netd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871240    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-kubelet\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.870143    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871240    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-daemon-config\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.872338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-host\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-netns\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cnibin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-run\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871790    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-socket-dir-parent\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871815    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-conf\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-system-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-hostroot\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-modprobe-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-lib-modules\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-sysctl-d\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-cni-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-var-lib-kubelet\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.871980    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-multus-conf-dir\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.872012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-run-multus-certs\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.872035    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13e8d794-ed88-440f-b2fe-e5f36f718e5b-host-var-lib-cni-bin\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.873292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.872485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13e8d794-ed88-440f-b2fe-e5f36f718e5b-cni-binary-copy\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.876919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.876531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-etc-tuned\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.880418 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.880027    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-tmp\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.888520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.888498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6d6g\" (UniqueName: \"kubernetes.io/projected/79e02a37-6bf8-4e1c-8a0c-ddae1d676359-kube-api-access-z6d6g\") pod \"tuned-85dn7\" (UID: \"79e02a37-6bf8-4e1c-8a0c-ddae1d676359\") " pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:19.891027 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.891002    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wcr\" (UniqueName: \"kubernetes.io/projected/13e8d794-ed88-440f-b2fe-e5f36f718e5b-kube-api-access-s4wcr\") pod \"multus-6fjrr\" (UID: \"13e8d794-ed88-440f-b2fe-e5f36f718e5b\") " pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:19.971377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqf92\" (UniqueName: \"kubernetes.io/projected/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-kube-api-access-qqf92\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.971377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-etc-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-tmp-dir\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-cnibin\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4969518-540b-43c6-af49-9166938fd718-host-slash\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-etc-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971487    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-konnectivity-ca\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4969518-540b-43c6-af49-9166938fd718-host-slash\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971556    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-cnibin\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-kubelet\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971602    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-agent-certs\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971617    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-kubelet\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.971632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971642    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4969518-540b-43c6-af49-9166938fd718-iptables-alerter-script\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971682    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-node-log\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-node-log\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-script-lib\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qqv\" (UniqueName: \"kubernetes.io/projected/5b1b1875-e67e-49de-9073-0333de341328-kube-api-access-z6qqv\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971819    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-device-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-netd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-config\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b1b1875-e67e-49de-9073-0333de341328-serviceca\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971917    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-device-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-tmp-dir\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.971994    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-log-socket\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972000    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-netd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972060    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-registration-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972087    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-log-socket\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-system-cni-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/956af29a-af32-4d5a-8eaa-ce00d3b27996-system-cni-dir\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-netns\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/65f66ff2-761b-4503-be4d-8bfb9e55536e-kube-api-access-wvlng\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972195    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-kubelet-config\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972208    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972249    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-slash\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-bin\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthgj\" (UniqueName: \"kubernetes.io/projected/a4969518-540b-43c6-af49-9166938fd718-kube-api-access-nthgj\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972348    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-hosts-file\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.972920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b1b1875-e67e-49de-9073-0333de341328-host\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972418    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-systemd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwhg\" (UniqueName: \"kubernetes.io/projected/8264c391-d5f3-44d2-b6cc-e00960c29712-kube-api-access-kbwhg\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4h7\" (UniqueName: \"kubernetes.io/projected/956af29a-af32-4d5a-8eaa-ce00d3b27996-kube-api-access-mr4h7\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-systemd-units\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-env-overrides\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972530    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-sys-fs\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd313698-c628-4132-8a0e-bbcc55063c11-ovn-node-metrics-cert\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972575    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-script-lib\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-socket-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-var-lib-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.973892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-ovn\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplnk\" (UniqueName: \"kubernetes.io/projected/dd313698-c628-4132-8a0e-bbcc55063c11-kube-api-access-wplnk\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-dbus\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.972912    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-dbus\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4969518-540b-43c6-af49-9166938fd718-iptables-alerter-script\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.973009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:20.472985843 +0000 UTC m=+3.103573646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-ovnkube-config\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973138    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-var-lib-openvswitch\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-ovn\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-registration-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-slash\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-konnectivity-ca\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.972167    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-run-netns\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973307    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-socket-dir\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c195e29-9ded-4d92-bebe-429d6354bd70-kubelet-config\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:19.974687 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.973373    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.973464    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:20.473444793 +0000 UTC m=+3.104032569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973527    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973535    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-host-cni-bin\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973619    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-run-systemd\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd313698-c628-4132-8a0e-bbcc55063c11-env-overrides\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd313698-c628-4132-8a0e-bbcc55063c11-systemd-units\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973828    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-hosts-file\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b1b1875-e67e-49de-9073-0333de341328-host\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.973943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/956af29a-af32-4d5a-8eaa-ce00d3b27996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.974301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b1b1875-e67e-49de-9073-0333de341328-serviceca\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.975364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.974426    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8264c391-d5f3-44d2-b6cc-e00960c29712-sys-fs\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.976361 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.976336    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd313698-c628-4132-8a0e-bbcc55063c11-ovn-node-metrics-cert\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:19.976503 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.976489    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c-agent-certs\") pod \"konnectivity-agent-qs7x4\" (UID: \"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c\") " pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:19.979773 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.979739    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:19.979974 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.979768    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:19.979974 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.979806    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:19.979974 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:19.979862    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:20.479846631 +0000 UTC m=+3.110434396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:19.981052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.981027    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqf92\" (UniqueName: \"kubernetes.io/projected/4dc63ba9-7b3b-4eaf-9900-4be507b168b7-kube-api-access-qqf92\") pod \"node-resolver-gqw7g\" (UID: \"4dc63ba9-7b3b-4eaf-9900-4be507b168b7\") " pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:19.981984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.981509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qqv\" (UniqueName: \"kubernetes.io/projected/5b1b1875-e67e-49de-9073-0333de341328-kube-api-access-z6qqv\") pod \"node-ca-bm5b2\" (UID: \"5b1b1875-e67e-49de-9073-0333de341328\") " pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:19.984280 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.984256    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/65f66ff2-761b-4503-be4d-8bfb9e55536e-kube-api-access-wvlng\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:19.984473 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.984453    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthgj\" (UniqueName: \"kubernetes.io/projected/a4969518-540b-43c6-af49-9166938fd718-kube-api-access-nthgj\") pod \"iptables-alerter-kcmhz\" (UID: \"a4969518-540b-43c6-af49-9166938fd718\") " pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:19.988310 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.988275    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4h7\" (UniqueName: \"kubernetes.io/projected/956af29a-af32-4d5a-8eaa-ce00d3b27996-kube-api-access-mr4h7\") pod \"multus-additional-cni-plugins-d6xx4\" (UID: \"956af29a-af32-4d5a-8eaa-ce00d3b27996\") " pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:19.988405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.988364    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwhg\" (UniqueName: \"kubernetes.io/projected/8264c391-d5f3-44d2-b6cc-e00960c29712-kube-api-access-kbwhg\") pod \"aws-ebs-csi-driver-node-lgjkx\" (UID: \"8264c391-d5f3-44d2-b6cc-e00960c29712\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:19.989492 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:19.989472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplnk\" (UniqueName: \"kubernetes.io/projected/dd313698-c628-4132-8a0e-bbcc55063c11-kube-api-access-wplnk\") pod \"ovnkube-node-snxqt\" (UID: \"dd313698-c628-4132-8a0e-bbcc55063c11\") " pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:20.071296 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.071240    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6fjrr"
Sep 09 12:11:20.079695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.079674    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85dn7"
Sep 09 12:11:20.099034 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.099004    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:20.103229 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.103201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gqw7g"
Sep 09 12:11:20.111322 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.111289    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bm5b2"
Sep 09 12:11:20.124364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.124296    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6xx4"
Sep 09 12:11:20.133510 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.133484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:20.147518 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.147489    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kcmhz"
Sep 09 12:11:20.153574 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.153543    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx"
Sep 09 12:11:20.401208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.401115    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:20.439295 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.439265    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 12:11:20.476975 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.476929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:20.477161 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.477037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:20.477161 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.477123    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:20.477314 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.477167    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:20.477314 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.477195    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:21.477179389 +0000 UTC m=+4.107767149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:20.477314 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.477224    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:21.477205458 +0000 UTC m=+4.107793237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:20.578305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.578269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:20.578511 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.578486    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:20.578570 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.578520    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:20.578570 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.578535    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:20.578635 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.578609    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:21.578588708 +0000 UTC m=+4.209176467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:20.839700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.839656    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:06:20 +0000 UTC" deadline="2028-04-29 02:14:34.736726019 +0000 UTC"
Sep 09 12:11:20.839700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.839695    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14342h3m13.897035289s"
Sep 09 12:11:20.895711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:20.895671    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:20.895840 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:20.895809    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:20.990485 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.990455    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8264c391_d5f3_44d2_b6cc_e00960c29712.slice/crio-e3cdc769aa7d121caa5b4377b2515757dc9bb8fc0a677357607981bb512b828d WatchSource:0}: Error finding container e3cdc769aa7d121caa5b4377b2515757dc9bb8fc0a677357607981bb512b828d: Status 404 returned error can't find the container with id e3cdc769aa7d121caa5b4377b2515757dc9bb8fc0a677357607981bb512b828d
Sep 09 12:11:20.991139 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.991064    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956af29a_af32_4d5a_8eaa_ce00d3b27996.slice/crio-a3d8ca5d97e0128aeb4fef61faa4ed2a6129f310351a7c184f22ddc3c433902d WatchSource:0}: Error finding container a3d8ca5d97e0128aeb4fef61faa4ed2a6129f310351a7c184f22ddc3c433902d: Status 404 returned error can't find the container with id a3d8ca5d97e0128aeb4fef61faa4ed2a6129f310351a7c184f22ddc3c433902d
Sep 09 12:11:20.993825 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.993801    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1f2f26_9ca4_4350_b0c1_e1cbcd1d972c.slice/crio-d05d38497557ae4f890551f4dfb99846db7d91e3b0c8a6aedf0c6d3f391e8d53 WatchSource:0}: Error finding container d05d38497557ae4f890551f4dfb99846db7d91e3b0c8a6aedf0c6d3f391e8d53: Status 404 returned error can't find the container with id d05d38497557ae4f890551f4dfb99846db7d91e3b0c8a6aedf0c6d3f391e8d53
Sep 09 12:11:20.996587 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.996537    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4969518_540b_43c6_af49_9166938fd718.slice/crio-d036fb157b2141f782e9b5e7c45623c26afff372212e0842e3d434294ccf7b99 WatchSource:0}: Error finding container d036fb157b2141f782e9b5e7c45623c26afff372212e0842e3d434294ccf7b99: Status 404 returned error can't find the container with id d036fb157b2141f782e9b5e7c45623c26afff372212e0842e3d434294ccf7b99
Sep 09 12:11:20.998249 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.998229    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8d794_ed88_440f_b2fe_e5f36f718e5b.slice/crio-d58e9c59e42236fc9cceb5e3aa4e27edf5eaf04e0568c49e18d3c3c1e56782a1 WatchSource:0}: Error finding container d58e9c59e42236fc9cceb5e3aa4e27edf5eaf04e0568c49e18d3c3c1e56782a1: Status 404 returned error can't find the container with id d58e9c59e42236fc9cceb5e3aa4e27edf5eaf04e0568c49e18d3c3c1e56782a1
Sep 09 12:11:20.999352 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:20.999323    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e02a37_6bf8_4e1c_8a0c_ddae1d676359.slice/crio-511cc2c5098aa5899eb27af5282419edc926e0f6af5a19ed7e2d560967b2ad3a WatchSource:0}: Error finding container 511cc2c5098aa5899eb27af5282419edc926e0f6af5a19ed7e2d560967b2ad3a: Status 404 returned error can't find the container with id 511cc2c5098aa5899eb27af5282419edc926e0f6af5a19ed7e2d560967b2ad3a
Sep 09 12:11:21.000760 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:21.000351    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd313698_c628_4132_8a0e_bbcc55063c11.slice/crio-b99f4f694b8a76162232013dc3ae67cbf487ed4d37c3decf199f85cf9f320a2a WatchSource:0}: Error finding container b99f4f694b8a76162232013dc3ae67cbf487ed4d37c3decf199f85cf9f320a2a: Status 404 returned error can't find the container with id b99f4f694b8a76162232013dc3ae67cbf487ed4d37c3decf199f85cf9f320a2a
Sep 09 12:11:21.001828 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:21.001805    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc63ba9_7b3b_4eaf_9900_4be507b168b7.slice/crio-beb3dc02f464ba139acf461ce171141d058c18e6669e6c163ca9769ca426c936 WatchSource:0}: Error finding container beb3dc02f464ba139acf461ce171141d058c18e6669e6c163ca9769ca426c936: Status 404 returned error can't find the container with id beb3dc02f464ba139acf461ce171141d058c18e6669e6c163ca9769ca426c936
Sep 09 12:11:21.002557 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:21.002531    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1b1875_e67e_49de_9073_0333de341328.slice/crio-6b4eca95d1bb72f566d343597159c61c28ae74f5e14ec35726e97938da68d860 WatchSource:0}: Error finding container 6b4eca95d1bb72f566d343597159c61c28ae74f5e14ec35726e97938da68d860: Status 404 returned error can't find the container with id 6b4eca95d1bb72f566d343597159c61c28ae74f5e14ec35726e97938da68d860
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.485514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.485608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.485743    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.485806    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:23.485791482 +0000 UTC m=+6.116379252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.486203    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:21.486371 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.486260    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:23.486240432 +0000 UTC m=+6.116828204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:21.586558 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.586512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:21.586760 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.586745    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:21.586827 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.586771    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:21.586827 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.586785    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:21.586937 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.586848    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:23.586829405 +0000 UTC m=+6.217417177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:21.840519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.840421    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 12:06:20 +0000 UTC" deadline="2028-02-25 16:17:47.645663198 +0000 UTC"
Sep 09 12:11:21.840519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.840467    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12820h6m25.805201592s"
Sep 09 12:11:21.902198 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.902165    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:21.902380 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.902333    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:21.902857 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.902699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:21.902857 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:21.902803    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:21.911788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.911753    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"b99f4f694b8a76162232013dc3ae67cbf487ed4d37c3decf199f85cf9f320a2a"}
Sep 09 12:11:21.917220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.917188    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qs7x4" event={"ID":"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c","Type":"ContainerStarted","Data":"d05d38497557ae4f890551f4dfb99846db7d91e3b0c8a6aedf0c6d3f391e8d53"}
Sep 09 12:11:21.920867 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.920837    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerStarted","Data":"a3d8ca5d97e0128aeb4fef61faa4ed2a6129f310351a7c184f22ddc3c433902d"}
Sep 09 12:11:21.928515 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.928485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6fjrr" event={"ID":"13e8d794-ed88-440f-b2fe-e5f36f718e5b","Type":"ContainerStarted","Data":"d58e9c59e42236fc9cceb5e3aa4e27edf5eaf04e0568c49e18d3c3c1e56782a1"}
Sep 09 12:11:21.931776 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.931743    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85dn7" event={"ID":"79e02a37-6bf8-4e1c-8a0c-ddae1d676359","Type":"ContainerStarted","Data":"511cc2c5098aa5899eb27af5282419edc926e0f6af5a19ed7e2d560967b2ad3a"}
Sep 09 12:11:21.934884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.934854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kcmhz" event={"ID":"a4969518-540b-43c6-af49-9166938fd718","Type":"ContainerStarted","Data":"d036fb157b2141f782e9b5e7c45623c26afff372212e0842e3d434294ccf7b99"}
Sep 09 12:11:21.939242 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.939214    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx" event={"ID":"8264c391-d5f3-44d2-b6cc-e00960c29712","Type":"ContainerStarted","Data":"e3cdc769aa7d121caa5b4377b2515757dc9bb8fc0a677357607981bb512b828d"}
Sep 09 12:11:21.945025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.944992    2556 generic.go:356] "Generic (PLEG): container finished" podID="c9542204e0b5b72ab121300d1ff50143" containerID="a338918e3444e824e922052e4223cdd1ec2dcad955b33e5d6b5f1dad1a7a84d3" exitCode=0
Sep 09 12:11:21.945132 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.945088    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal" event={"ID":"c9542204e0b5b72ab121300d1ff50143","Type":"ContainerDied","Data":"a338918e3444e824e922052e4223cdd1ec2dcad955b33e5d6b5f1dad1a7a84d3"}
Sep 09 12:11:21.949686 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.949656    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal" event={"ID":"707ed0fa4514c61dc617937a9e19c4ec","Type":"ContainerStarted","Data":"a85bd348d1ba41a13d1ec229c90f81344c3a209262a7eb6d7069135c5a9632c8"}
Sep 09 12:11:21.958351 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.958307    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bm5b2" event={"ID":"5b1b1875-e67e-49de-9073-0333de341328","Type":"ContainerStarted","Data":"6b4eca95d1bb72f566d343597159c61c28ae74f5e14ec35726e97938da68d860"}
Sep 09 12:11:21.961288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:21.961261    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gqw7g" event={"ID":"4dc63ba9-7b3b-4eaf-9900-4be507b168b7","Type":"ContainerStarted","Data":"beb3dc02f464ba139acf461ce171141d058c18e6669e6c163ca9769ca426c936"}
Sep 09 12:11:22.899588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:22.899021    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:22.899588 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:22.899175    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:22.979674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:22.979636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal" event={"ID":"c9542204e0b5b72ab121300d1ff50143","Type":"ContainerStarted","Data":"f9aeebe9986be9f2d0eecae2fa5c83803f0d7af2d41fac5481bf2714aa55d542"}
Sep 09 12:11:23.008105 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.007401    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-117.ec2.internal" podStartSLOduration=4.007377072 podStartE2EDuration="4.007377072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:21.983247116 +0000 UTC m=+4.613834896" watchObservedRunningTime="2026-09-09 12:11:23.007377072 +0000 UTC m=+5.637964856"
Sep 09 12:11:23.515938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.515425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:23.515938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.515519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:23.515938 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.515646    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:23.515938 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.515710    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:27.515690915 +0000 UTC m=+10.146278672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:23.516305 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.516099    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:23.516305 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.516146    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:27.516130145 +0000 UTC m=+10.146717899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:23.616960 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.616231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:23.616960 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.616426    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:23.616960 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.616451    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:23.616960 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.616464    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:23.616960 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.616528    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:27.616507969 +0000 UTC m=+10.247095740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:23.896481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.896392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:23.896646 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.896551    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:23.899374 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:23.899339    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:23.899493 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:23.899453    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:24.896237 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:24.895676    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:24.896237 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:24.895810    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:25.896575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:25.896544    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:25.897078 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:25.896679    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:25.899812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:25.896507    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:25.899812 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:25.899757    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:26.896318 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:26.896288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:26.896597 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:26.896412    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:27.550688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:27.550761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.550917    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.550981    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:35.550960778 +0000 UTC m=+18.181548536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.551375    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:27.551465 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.551425    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:35.551409297 +0000 UTC m=+18.181997057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:27.651926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:27.651885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:27.652089 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.652056    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:27.652089 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.652072    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:27.652089 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.652081    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:27.652359 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.652134    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:35.652115333 +0000 UTC m=+18.282703109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:27.898177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:27.896655    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:27.898177 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.897016    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:27.898177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:27.897383    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:27.898177 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:27.897483    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:28.895886 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:28.895844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:28.896327 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:28.895991    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:29.898850 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:29.898160    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:29.898850 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:29.898313    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:29.898850 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:29.898694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:29.898850 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:29.898786    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:30.896016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:30.895980    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:30.897265 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:30.896592    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:31.896527 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:31.896497    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:31.896963 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:31.896622    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:31.896963 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:31.896733    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:31.896963 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:31.896817    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:32.895879 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:32.895837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:32.896064 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:32.895974    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:33.896100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:33.896059    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:33.896571 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:33.896209    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:33.896571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:33.896262    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:33.896571 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:33.896377    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:34.895775 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:34.895742    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:34.895934 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:34.895862    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:35.609809 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:35.609772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:35.610263 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:35.609848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:35.610263 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.609946    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:35.610263 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.609946    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:35.610263 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.610009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:51.60999543 +0000 UTC m=+34.240583187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:35.610263 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.610024    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:11:51.610016821 +0000 UTC m=+34.240604574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:35.711189 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:35.711149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:35.711387 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.711345    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:35.711387 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.711372    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:35.711387 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.711386    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:35.711542 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.711455    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:51.711434626 +0000 UTC m=+34.342022397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:35.899492 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:35.899414    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:35.899492 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:35.899441    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:35.899723 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.899527    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:35.899723 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:35.899674    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:36.896290 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:36.896255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:36.896748 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:36.896386    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:37.897191 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:37.897156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:37.897627 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:37.897298    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:37.897627 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:37.897359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:37.897627 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:37.897475    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:38.896035 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:38.896007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:38.896170 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:38.896142    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:39.015262 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.015220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85dn7" event={"ID":"79e02a37-6bf8-4e1c-8a0c-ddae1d676359","Type":"ContainerStarted","Data":"92a720f12c0e181688a309ff2188621a9ee7ca78eb527fef3ba3b8b8f43ca5a2"}
Sep 09 12:11:39.017483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.017462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx" event={"ID":"8264c391-d5f3-44d2-b6cc-e00960c29712","Type":"ContainerStarted","Data":"11eb6d93f1e85dbd69a7f8d341409fff9a647be89f8626bf8d64c68f27cfd7c6"}
Sep 09 12:11:39.019211 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.019192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bm5b2" event={"ID":"5b1b1875-e67e-49de-9073-0333de341328","Type":"ContainerStarted","Data":"fb2c00c64b48da5ffcfad1f604f2a8398fdf16635891ee5bff477a85811c3733"}
Sep 09 12:11:39.020721 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.020675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gqw7g" event={"ID":"4dc63ba9-7b3b-4eaf-9900-4be507b168b7","Type":"ContainerStarted","Data":"92aa4521acfb651f4aba0194f6bbd0d20b60db2dc6aed0d0ac138ad115e1da01"}
Sep 09 12:11:39.022282 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.022089    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qs7x4" event={"ID":"fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c","Type":"ContainerStarted","Data":"14472be5a6a0c77d260ed2ba9b66892b059a572b877853a000950022dc795a9c"}
Sep 09 12:11:39.034198 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.034150    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-117.ec2.internal" podStartSLOduration=20.034133886 podStartE2EDuration="20.034133886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:23.007824915 +0000 UTC m=+5.638412695" watchObservedRunningTime="2026-09-09 12:11:39.034133886 +0000 UTC m=+21.664721663"
Sep 09 12:11:39.034555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.034533    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-85dn7" podStartSLOduration=3.286450843 podStartE2EDuration="21.03452864s" totalImagesPullingTime="17.748077797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:39.033822765 +0000 UTC m=+21.664410541" watchObservedRunningTime="2026-09-09 12:11:39.03452864 +0000 UTC m=+21.665116450"
Sep 09 12:11:39.051654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.051614    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qs7x4" podStartSLOduration=3.763341069 podStartE2EDuration="21.051603632s" totalImagesPullingTime="17.288262563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:39.051268003 +0000 UTC m=+21.681855775" watchObservedRunningTime="2026-09-09 12:11:39.051603632 +0000 UTC m=+21.682191408"
Sep 09 12:11:39.066605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.066565    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bm5b2" podStartSLOduration=3.787178791 podStartE2EDuration="21.066555471s" totalImagesPullingTime="17.27937668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:39.066325293 +0000 UTC m=+21.696913070" watchObservedRunningTime="2026-09-09 12:11:39.066555471 +0000 UTC m=+21.697143243"
Sep 09 12:11:39.896337 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.896065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:39.896616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:39.896078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:39.896616 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:39.896454    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:39.896616 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:39.896483    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:40.025311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.025273    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6fjrr" event={"ID":"13e8d794-ed88-440f-b2fe-e5f36f718e5b","Type":"ContainerStarted","Data":"9d2fc8ef59f993a82bbb000bfe66076d94826de3577766cbc8275d98d5b9b613"}
Sep 09 12:11:40.028219 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028201    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:11:40.028585 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028559    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd313698-c628-4132-8a0e-bbcc55063c11" containerID="e22be1f042073676c092544962a2d3b6ec906746ccfa5945d2bfa20de36c7e09" exitCode=1
Sep 09 12:11:40.028635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"3935274201f2eeb660d6eb0d17ac2a99273db193ede59bf9ce582f3862beea61"}
Sep 09 12:11:40.028708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028693    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"a05d769f31284c52418946a667a84bfb30c56b76975a7263b961cc34f1e141f0"}
Sep 09 12:11:40.028762 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"03b89095fef67ed0ac415b95ef8665a80de374d83ba18305d3d9804cf8cc560a"}
Sep 09 12:11:40.028813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"5ec736e979b6f4a43c6bbfacf22bedaf3b31c9a86dcdae0978007173dc6bf9fc"}
Sep 09 12:11:40.028888 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerDied","Data":"e22be1f042073676c092544962a2d3b6ec906746ccfa5945d2bfa20de36c7e09"}
Sep 09 12:11:40.028925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.028916    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"179e47445efab1b48473d89947fc772eebc77278af3c13bdb0c056b07471502d"}
Sep 09 12:11:40.030269 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.030232    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="eccf0b63936a912a82df9276402331d248490f6850f09768aa3eb4db10855ca6" exitCode=0
Sep 09 12:11:40.030385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.030343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"eccf0b63936a912a82df9276402331d248490f6850f09768aa3eb4db10855ca6"}
Sep 09 12:11:40.048978 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.048936    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gqw7g" podStartSLOduration=4.337275077 podStartE2EDuration="22.048922628s" totalImagesPullingTime="17.711647551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:39.08140501 +0000 UTC m=+21.711992787" watchObservedRunningTime="2026-09-09 12:11:40.048922628 +0000 UTC m=+22.679510451"
Sep 09 12:11:40.049111 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.049087    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6fjrr" podStartSLOduration=4.292963608 podStartE2EDuration="22.049078153s" totalImagesPullingTime="17.756114545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:40.048636143 +0000 UTC m=+22.679223919" watchObservedRunningTime="2026-09-09 12:11:40.049078153 +0000 UTC m=+22.679665940"
Sep 09 12:11:40.206355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.206279    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:40.279551 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.279326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:40.371937 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.371913    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 12:11:40.433132 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.433099    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:40.496029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.495999    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:40.832120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.831992    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T12:11:40.37192961Z","UUID":"8c2cf4ce-0a07-4312-9f19-5fa8fe839bfb","Handler":null,"Name":"","Endpoint":""}
Sep 09 12:11:40.833331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.833311    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 12:11:40.833331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.833338    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 12:11:40.895774 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:40.895746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:40.895946 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:40.895855    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:41.034220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.034183    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kcmhz" event={"ID":"a4969518-540b-43c6-af49-9166938fd718","Type":"ContainerStarted","Data":"2dd74feaca9bf14967f5d9793514f17a927d02adc1c12379fb055f6dfdb5125b"}
Sep 09 12:11:41.036038 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.036016    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx" event={"ID":"8264c391-d5f3-44d2-b6cc-e00960c29712","Type":"ContainerStarted","Data":"adf015e3fb70fda41cbdaf8c7e7e87e15122146dfc2aa2a4a85f535f64b3bf72"}
Sep 09 12:11:41.257216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.257177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:41.385181 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.385148    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:41.896393 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.896315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:41.896393 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:41.896338    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:41.896599 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:41.896425    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:41.896599 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:41.896548    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:42.042630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.042608    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:11:42.043321 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.043294    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"920cc4f64cb46235e3547e1f016552a4ce5f823415ff8ce7d5df31e9af20145c"}
Sep 09 12:11:42.221494 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.221446    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:42.222112 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.222086    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:42.239208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.239153    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kcmhz" podStartSLOduration=6.488364824 podStartE2EDuration="24.239136859s" totalImagesPullingTime="17.750772035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:41.055482314 +0000 UTC m=+23.686070090" watchObservedRunningTime="2026-09-09 12:11:42.239136859 +0000 UTC m=+24.869724635"
Sep 09 12:11:42.896412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:42.896376    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:42.896586 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:42.896506    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:43.050471 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.050436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx" event={"ID":"8264c391-d5f3-44d2-b6cc-e00960c29712","Type":"ContainerStarted","Data":"d00e06e6c50e75ac99989f5ef0ab780f21025b05b6521a069743f7b2938d3e34"}
Sep 09 12:11:43.050855 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.050502    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:43.050969 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.050954    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qs7x4"
Sep 09 12:11:43.068220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.068172    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgjkx" podStartSLOduration=4.542734805 podStartE2EDuration="25.068149365s" totalImagesPullingTime="20.52541456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:43.068017637 +0000 UTC m=+25.698605425" watchObservedRunningTime="2026-09-09 12:11:43.068149365 +0000 UTC m=+25.698737138"
Sep 09 12:11:43.137112 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.137081    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log" limit=1024
Sep 09 12:11:43.896338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.896315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:43.896458 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:43.896318    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:43.896510 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:43.896451    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:43.896559 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:43.896506    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:44.054976 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:44.054719    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:11:44.055408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:44.055343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"c22d389a4e623a0e73a92e49a30a314e8f70dd2a1d4a8a56d870dec0f6556c11"}
Sep 09 12:11:44.055900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:44.055880    2556 scope.go:117] "RemoveContainer" containerID="e22be1f042073676c092544962a2d3b6ec906746ccfa5945d2bfa20de36c7e09"
Sep 09 12:11:44.895868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:44.895837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:44.896021 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:44.895951    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:45.062474 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.062437    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:11:45.063053 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.062985    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" event={"ID":"dd313698-c628-4132-8a0e-bbcc55063c11","Type":"ContainerStarted","Data":"6267de24b010e8c8dba49c6453a736a50e86c8d00bdb455f80adaed1a9b5f79d"}
Sep 09 12:11:45.063053 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.063018    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 12:11:45.063458 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.063436    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:45.063583 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.063516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:45.089272 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.089143    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:45.091384 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.091144    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:45.094889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.093951    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt" podStartSLOduration=9.305711034 podStartE2EDuration="27.093933175s" totalImagesPullingTime="17.788222141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:45.093887074 +0000 UTC m=+27.724474850" watchObservedRunningTime="2026-09-09 12:11:45.093933175 +0000 UTC m=+27.724520950"
Sep 09 12:11:45.173217 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.173129    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:11:45.896188 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.895846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:45.896188 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.895846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:45.896188 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:45.896139    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:45.896393 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:45.896289    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:45.901098 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.901076    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l7hxr"]
Sep 09 12:11:45.901212 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.901140    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gm6c"]
Sep 09 12:11:45.901427 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.901395    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wctrv"]
Sep 09 12:11:45.901505 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:45.901492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:45.901623 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:45.901603    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:46.066910 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:46.066880    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="5e4c9f7f22122f1d87cf2f6ad4c51b2f39c7610418a1c3eed81cdb7888735c61" exitCode=0
Sep 09 12:11:46.067487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:46.066980    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"5e4c9f7f22122f1d87cf2f6ad4c51b2f39c7610418a1c3eed81cdb7888735c61"}
Sep 09 12:11:46.067487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:46.067010    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:46.067487 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:46.067133    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:46.067487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:46.067187    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:46.067487 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:46.067278    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:47.070916 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:47.070837    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="6c4360e8f75a6f9f34dba986ab04ef871a8745d43bcbc69f7c90c4616e283a1e" exitCode=0
Sep 09 12:11:47.071278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:47.070914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"6c4360e8f75a6f9f34dba986ab04ef871a8745d43bcbc69f7c90c4616e283a1e"}
Sep 09 12:11:47.897070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:47.897040    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:47.897221 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:47.897132    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:47.897556 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:47.897443    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:47.897556 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:47.897520    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:47.897714 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:47.897527    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:47.897714 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:47.897639    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:49.078378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:49.078347    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="79866f32adc808e07e7b2450befc877022391c9675b7180840e39a5bdb54fa63" exitCode=0
Sep 09 12:11:49.078816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:49.078410    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"79866f32adc808e07e7b2450befc877022391c9675b7180840e39a5bdb54fa63"}
Sep 09 12:11:49.896107 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:49.896073    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:49.896309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:49.896119    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:49.896309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:49.896075    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:49.896309 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:49.896181    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gm6c" podUID="3673b08c-2a12-411f-a631-e8c142056089"
Sep 09 12:11:49.896469 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:49.896315    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hxr" podUID="7c195e29-9ded-4d92-bebe-429d6354bd70"
Sep 09 12:11:49.896469 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:49.896424    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wctrv" podUID="65f66ff2-761b-4503-be4d-8bfb9e55536e"
Sep 09 12:11:51.593703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.593673    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-117.ec2.internal" event="NodeReady"
Sep 09 12:11:51.594171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.593824    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 12:11:51.627860 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.627828    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:51.628046 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.627889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:51.628046 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.627994    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:51.628046 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.628010    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:51.628203 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.628059    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret podName:7c195e29-9ded-4d92-bebe-429d6354bd70 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:23.62803899 +0000 UTC m=+66.258626744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret") pod "global-pull-secret-syncer-l7hxr" (UID: "7c195e29-9ded-4d92-bebe-429d6354bd70") : object "kube-system"/"original-pull-secret" not registered
Sep 09 12:11:51.628203 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.628078    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs podName:65f66ff2-761b-4503-be4d-8bfb9e55536e nodeName:}" failed. No retries permitted until 2026-09-09 12:12:23.628068403 +0000 UTC m=+66.258656160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs") pod "network-metrics-daemon-wctrv" (UID: "65f66ff2-761b-4503-be4d-8bfb9e55536e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 12:11:51.671068 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.671042    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m5j95"]
Sep 09 12:11:51.674471 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.674428    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.679134 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.679104    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 12:11:51.679299 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.679265    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mksrl\""
Sep 09 12:11:51.679405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.679387    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 12:11:51.690094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.688393    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vqnpx"]
Sep 09 12:11:51.692394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.692349    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.696882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.696860    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 12:11:51.696992 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.696965    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 12:11:51.697062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.697012    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xqhh6\""
Sep 09 12:11:51.697604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.697448    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 12:11:51.703785 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.703764    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5j95"]
Sep 09 12:11:51.714023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.714002    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqnpx"]
Sep 09 12:11:51.728653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.728632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:51.728794 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.728778    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 12:11:51.728833 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.728800    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 12:11:51.728833 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.728809    2556 projected.go:196] Error preparing data for projected volume kube-api-access-mpqwp for pod openshift-network-diagnostics/network-check-target-5gm6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:51.728908 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.728858    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp podName:3673b08c-2a12-411f-a631-e8c142056089 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:23.728844909 +0000 UTC m=+66.359432667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mpqwp" (UniqueName: "kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp") pod "network-check-target-5gm6c" (UID: "3673b08c-2a12-411f-a631-e8c142056089") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 12:11:51.829860 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.829820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcssl\" (UniqueName: \"kubernetes.io/projected/0878ddec-9390-4909-b2ea-4cb57aea3698-kube-api-access-pcssl\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.829860 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.829866    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d79601f5-c28a-4c83-8c21-57e7e1d31668-cert\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.830089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.829889    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0878ddec-9390-4909-b2ea-4cb57aea3698-config-volume\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.830089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.829931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0878ddec-9390-4909-b2ea-4cb57aea3698-tmp-dir\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.830089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.830031    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.830089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.830079    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2bm\" (UniqueName: \"kubernetes.io/projected/d79601f5-c28a-4c83-8c21-57e7e1d31668-kube-api-access-lc2bm\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.895964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.895927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:11:51.896126 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.895999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:11:51.896126 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.896002    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:11:51.898604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.898422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:11:51.898604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.898498    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:11:51.898604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.898507    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 12:11:51.898604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.898537    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s86q\""
Sep 09 12:11:51.899069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.899046    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 12:11:51.899298 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.899282    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g5zgt\""
Sep 09 12:11:51.931074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0878ddec-9390-4909-b2ea-4cb57aea3698-tmp-dir\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2bm\" (UniqueName: \"kubernetes.io/projected/d79601f5-c28a-4c83-8c21-57e7e1d31668-kube-api-access-lc2bm\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.931203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcssl\" (UniqueName: \"kubernetes.io/projected/0878ddec-9390-4909-b2ea-4cb57aea3698-kube-api-access-pcssl\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d79601f5-c28a-4c83-8c21-57e7e1d31668-cert\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.931412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0878ddec-9390-4909-b2ea-4cb57aea3698-config-volume\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931453 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931433    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0878ddec-9390-4909-b2ea-4cb57aea3698-tmp-dir\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.931695    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0878ddec-9390-4909-b2ea-4cb57aea3698-config-volume\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.931806 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.931792    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:11:51.931863 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:51.931853    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:52.431838406 +0000 UTC m=+35.062426160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:11:51.936857 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.936722    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d79601f5-c28a-4c83-8c21-57e7e1d31668-cert\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:51.947179 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.947127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcssl\" (UniqueName: \"kubernetes.io/projected/0878ddec-9390-4909-b2ea-4cb57aea3698-kube-api-access-pcssl\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:51.947179 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:51.947168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2bm\" (UniqueName: \"kubernetes.io/projected/d79601f5-c28a-4c83-8c21-57e7e1d31668-kube-api-access-lc2bm\") pod \"ingress-canary-vqnpx\" (UID: \"d79601f5-c28a-4c83-8c21-57e7e1d31668\") " pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:52.000300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.000261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqnpx"
Sep 09 12:11:52.289103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.289042    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"]
Sep 09 12:11:52.293798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.293772    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"]
Sep 09 12:11:52.294083 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.294050    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.297451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.297430    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-867b7bd764-5qsj7"]
Sep 09 12:11:52.297550 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.297539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.298691 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.298670    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 12:11:52.300041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.299775    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 12:11:52.300041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.299776    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x5mnw\""
Sep 09 12:11:52.300041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.299820    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 12:11:52.300041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.299780    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 12:11:52.300372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.300144    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:11:52.300372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.300205    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 12:11:52.300728 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.300708    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pcrmj\""
Sep 09 12:11:52.300812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.300769    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 12:11:52.300869 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.300718    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 12:11:52.301065 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.301047    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqnpx"]
Sep 09 12:11:52.301143 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:52.301087    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79601f5_c28a_4c83_8c21_57e7e1d31668.slice/crio-8d93c9bf16590ce2fd9ba924b4fdb0020c190e708573eb2f7d32258710c92258 WatchSource:0}: Error finding container 8d93c9bf16590ce2fd9ba924b4fdb0020c190e708573eb2f7d32258710c92258: Status 404 returned error can't find the container with id 8d93c9bf16590ce2fd9ba924b4fdb0020c190e708573eb2f7d32258710c92258
Sep 09 12:11:52.301206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.301146    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.304751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.304660    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 12:11:52.304751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.304667    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 12:11:52.304919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.304817    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-959qs\""
Sep 09 12:11:52.304919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.304856    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 12:11:52.305398 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.305377    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 12:11:52.305588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.305573    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 12:11:52.306300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.305755    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 12:11:52.306826 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.306796    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"]
Sep 09 12:11:52.307440 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.307421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"]
Sep 09 12:11:52.310592 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.310555    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-867b7bd764-5qsj7"]
Sep 09 12:11:52.380607 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.380570    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-75fzc"]
Sep 09 12:11:52.384442 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.384420    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.386986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.386957    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 12:11:52.387106 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.386994    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 12:11:52.387448 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.387428    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 12:11:52.389630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.388822    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-drslw\""
Sep 09 12:11:52.389630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.389021    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 12:11:52.392549 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.392527    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 12:11:52.394979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.394927    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-75fzc"]
Sep 09 12:11:52.436097 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnvx\" (UniqueName: \"kubernetes.io/projected/0ef14022-b373-4895-8eff-b16e98fa7029-kube-api-access-4mnvx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.436097 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwb9\" (UniqueName: \"kubernetes.io/projected/eb1847d7-9261-487a-be2c-972566991417-kube-api-access-gvwb9\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.436315 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.436315 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb1847d7-9261-487a-be2c-972566991417-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.436315 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-default-certificate\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.436315 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.436481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef14022-b373-4895-8eff-b16e98fa7029-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.436481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlxt\" (UniqueName: \"kubernetes.io/projected/63cc97ea-6218-4078-b500-bec5db94a85f-kube-api-access-nnlxt\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.436481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef14022-b373-4895-8eff-b16e98fa7029-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.436481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:52.436481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.436661 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.436481    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-stats-auth\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.436661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.436530    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:11:52.436661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.436582    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:53.436567248 +0000 UTC m=+36.067155004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:11:52.537045 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.536952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef14022-b373-4895-8eff-b16e98fa7029-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.537045 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.537045 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-stats-auth\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-tmp\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnvx\" (UniqueName: \"kubernetes.io/projected/0ef14022-b373-4895-8eff-b16e98fa7029-kube-api-access-4mnvx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwb9\" (UniqueName: \"kubernetes.io/projected/eb1847d7-9261-487a-be2c-972566991417-kube-api-access-gvwb9\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.537331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb1847d7-9261-487a-be2c-972566991417-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.537363    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.537436    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:53.037413664 +0000 UTC m=+35.668001418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537550    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-default-certificate\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537586    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/640e5fd6-5c1f-4b28-a14d-4cb59e947583-serving-cert\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-snapshots\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.537802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef14022-b373-4895-8eff-b16e98fa7029-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.538145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/eb1847d7-9261-487a-be2c-972566991417-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.538145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlxt\" (UniqueName: \"kubernetes.io/projected/63cc97ea-6218-4078-b500-bec5db94a85f-kube-api-access-nnlxt\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.538145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.537966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jcsm\" (UniqueName: \"kubernetes.io/projected/640e5fd6-5c1f-4b28-a14d-4cb59e947583-kube-api-access-7jcsm\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.538145 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.538021    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:53.038002283 +0000 UTC m=+35.668590058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:11:52.538145 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.538121    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:11:52.538434 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.538179    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef14022-b373-4895-8eff-b16e98fa7029-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.538434 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:52.538368    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:53.038351807 +0000 UTC m=+35.668939568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:11:52.540473 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.540447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef14022-b373-4895-8eff-b16e98fa7029-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.540653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.540635    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-stats-auth\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.540925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.540908    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-default-certificate\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.549794 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.549765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnvx\" (UniqueName: \"kubernetes.io/projected/0ef14022-b373-4895-8eff-b16e98fa7029-kube-api-access-4mnvx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xwht7\" (UID: \"0ef14022-b373-4895-8eff-b16e98fa7029\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.551588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.551561    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlxt\" (UniqueName: \"kubernetes.io/projected/63cc97ea-6218-4078-b500-bec5db94a85f-kube-api-access-nnlxt\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:52.552750 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.552727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwb9\" (UniqueName: \"kubernetes.io/projected/eb1847d7-9261-487a-be2c-972566991417-kube-api-access-gvwb9\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:52.618289 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.618230    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"
Sep 09 12:11:52.638531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/640e5fd6-5c1f-4b28-a14d-4cb59e947583-serving-cert\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.638650 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-snapshots\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.638650 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638577    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jcsm\" (UniqueName: \"kubernetes.io/projected/640e5fd6-5c1f-4b28-a14d-4cb59e947583-kube-api-access-7jcsm\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.638650 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638619    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.638813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-tmp\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.638813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.638676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.639230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.639208    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.639419 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.639398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-tmp\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.639922 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.639900    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640e5fd6-5c1f-4b28-a14d-4cb59e947583-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.640009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.639990    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/640e5fd6-5c1f-4b28-a14d-4cb59e947583-snapshots\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.641628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.641602    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/640e5fd6-5c1f-4b28-a14d-4cb59e947583-serving-cert\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.659585 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.659519    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jcsm\" (UniqueName: \"kubernetes.io/projected/640e5fd6-5c1f-4b28-a14d-4cb59e947583-kube-api-access-7jcsm\") pod \"insights-operator-76bcd6ccf5-75fzc\" (UID: \"640e5fd6-5c1f-4b28-a14d-4cb59e947583\") " pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.695889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.695860    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc"
Sep 09 12:11:52.893672 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.893592    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7"]
Sep 09 12:11:52.897469 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:52.897439    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef14022_b373_4895_8eff_b16e98fa7029.slice/crio-571b19e3f246257d1e7260539b28048162e52ca3a8b3d24b78cca283f4d26d77 WatchSource:0}: Error finding container 571b19e3f246257d1e7260539b28048162e52ca3a8b3d24b78cca283f4d26d77: Status 404 returned error can't find the container with id 571b19e3f246257d1e7260539b28048162e52ca3a8b3d24b78cca283f4d26d77
Sep 09 12:11:52.976875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:52.976834    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-75fzc"]
Sep 09 12:11:52.979661 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:11:52.979623    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640e5fd6_5c1f_4b28_a14d_4cb59e947583.slice/crio-dc9996107298fbc06b9035790b960465d2b45c96c6ecda571b689625cbeeb0db WatchSource:0}: Error finding container dc9996107298fbc06b9035790b960465d2b45c96c6ecda571b689625cbeeb0db: Status 404 returned error can't find the container with id dc9996107298fbc06b9035790b960465d2b45c96c6ecda571b689625cbeeb0db
Sep 09 12:11:53.042335 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.042286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:53.042507 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.042348    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:53.042507 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.042440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:53.042507 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.042492    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:54.042468318 +0000 UTC m=+36.673056087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:11:53.042661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.042515    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:11:53.042661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.042530    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:53.042661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.042571    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:54.042554707 +0000 UTC m=+36.673142477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:11:53.042661 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.042589    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:54.042580066 +0000 UTC m=+36.673167823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:53.090199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.090157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc" event={"ID":"640e5fd6-5c1f-4b28-a14d-4cb59e947583","Type":"ContainerStarted","Data":"dc9996107298fbc06b9035790b960465d2b45c96c6ecda571b689625cbeeb0db"}
Sep 09 12:11:53.091500 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.091474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7" event={"ID":"0ef14022-b373-4895-8eff-b16e98fa7029","Type":"ContainerStarted","Data":"571b19e3f246257d1e7260539b28048162e52ca3a8b3d24b78cca283f4d26d77"}
Sep 09 12:11:53.092849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.092799    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqnpx" event={"ID":"d79601f5-c28a-4c83-8c21-57e7e1d31668","Type":"ContainerStarted","Data":"8d93c9bf16590ce2fd9ba924b4fdb0020c190e708573eb2f7d32258710c92258"}
Sep 09 12:11:53.446885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:53.446851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:53.447078 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.447009    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:11:53.447078 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:53.447077    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:55.44705766 +0000 UTC m=+38.077645428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:11:54.052914 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:54.052879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:54.052935    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:54.052959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:54.053059    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:54.053081    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:56.053063312 +0000 UTC m=+38.683651086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:54.053091    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:54.053104    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:11:56.0530945 +0000 UTC m=+38.683682259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:11:54.053630 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:54.053180    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:56.053158877 +0000 UTC m=+38.683746642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:55.465721 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:55.465670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:55.466149 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:55.465839    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:11:55.466149 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:55.465914    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:11:59.46589195 +0000 UTC m=+42.096479712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:11:56.070479 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:56.070427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:11:56.070651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:56.070511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:56.070651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:56.070548    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:11:56.070651 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:56.070617    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:56.070810 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:56.070664    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:11:56.070810 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:56.070702    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:00.070677476 +0000 UTC m=+42.701265249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:11:56.070810 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:56.070726    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:00.070715813 +0000 UTC m=+42.701303572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:11:56.070810 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:56.070762    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:00.070746023 +0000 UTC m=+42.701333791 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:11:58.112741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.112524    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7" event={"ID":"0ef14022-b373-4895-8eff-b16e98fa7029","Type":"ContainerStarted","Data":"beed1bca0cfd04c6c83e558a6d31f2c9921ef8c047cb670d4703fb4ff383a8be"}
Sep 09 12:11:58.114248 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.114214    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqnpx" event={"ID":"d79601f5-c28a-4c83-8c21-57e7e1d31668","Type":"ContainerStarted","Data":"572f0317577dee3c8490c7b40fe52ad67ea8e173d58c2498330e3aad47c93c3a"}
Sep 09 12:11:58.117127 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.117107    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="abe8a4f6f44d9c0bc77f1873b0f53dc026a82d51bd5fc68800eaffaae882df10" exitCode=0
Sep 09 12:11:58.117266 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.117168    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"abe8a4f6f44d9c0bc77f1873b0f53dc026a82d51bd5fc68800eaffaae882df10"}
Sep 09 12:11:58.118812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.118777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc" event={"ID":"640e5fd6-5c1f-4b28-a14d-4cb59e947583","Type":"ContainerStarted","Data":"9ba37ac91d37e3c0875e962d166d176a630a8f3eb94d46cf575899b20d154cab"}
Sep 09 12:11:58.130342 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.130300    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7" podStartSLOduration=1.7201741579999998 podStartE2EDuration="6.13028685s" totalImagesPullingTime="4.410112692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:58.129379579 +0000 UTC m=+40.759967358" watchObservedRunningTime="2026-09-09 12:11:58.13028685 +0000 UTC m=+40.760874625"
Sep 09 12:11:58.146054 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.146008    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vqnpx" podStartSLOduration=2.151689417 podStartE2EDuration="7.145993458s" totalImagesPullingTime="4.994304041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:58.145521124 +0000 UTC m=+40.776108900" watchObservedRunningTime="2026-09-09 12:11:58.145993458 +0000 UTC m=+40.776581250"
Sep 09 12:11:58.166489 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.166440    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc" podStartSLOduration=1.843560198 podStartE2EDuration="6.166420476s" totalImagesPullingTime="4.322860278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:58.166296159 +0000 UTC m=+40.796883936" watchObservedRunningTime="2026-09-09 12:11:58.166420476 +0000 UTC m=+40.797008253"
Sep 09 12:11:58.679985 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.679948    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mdwth"]
Sep 09 12:11:58.704517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.704487    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mdwth"]
Sep 09 12:11:58.704668 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.704582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.706771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.706744    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 12:11:58.706981 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.706968    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lrqz2\""
Sep 09 12:11:58.707273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.707261    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 12:11:58.895144 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.895109    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012a3b9a-7354-42df-b198-2011231deceb-data-volume\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.895144 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.895147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.895401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.895169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.895401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.895187    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vm4\" (UniqueName: \"kubernetes.io/projected/012a3b9a-7354-42df-b198-2011231deceb-kube-api-access-49vm4\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.895401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.895279    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012a3b9a-7354-42df-b198-2011231deceb-crio-socket\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996308 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012a3b9a-7354-42df-b198-2011231deceb-data-volume\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996308 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996509 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996509 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vm4\" (UniqueName: \"kubernetes.io/projected/012a3b9a-7354-42df-b198-2011231deceb-kube-api-access-49vm4\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996509 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:58.996417    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:11:58.996509 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:58.996463    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:11:58.996509 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:58.996488    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:11:59.496472747 +0000 UTC m=+42.127060501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : configmap "kube-rbac-proxy" not found
Sep 09 12:11:58.996676 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:58.996517    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:11:59.496499654 +0000 UTC m=+42.127087427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:11:58.996676 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996540    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012a3b9a-7354-42df-b198-2011231deceb-crio-socket\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996676 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996661    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012a3b9a-7354-42df-b198-2011231deceb-data-volume\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:58.996787 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:58.996770    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012a3b9a-7354-42df-b198-2011231deceb-crio-socket\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:59.008808 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.008783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vm4\" (UniqueName: \"kubernetes.io/projected/012a3b9a-7354-42df-b198-2011231deceb-kube-api-access-49vm4\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:59.124439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.124407    2556 generic.go:356] "Generic (PLEG): container finished" podID="956af29a-af32-4d5a-8eaa-ce00d3b27996" containerID="528012d1df35cedc83f64efe1c6f3ef741be3221ad0b21469f4782377a1ecb5c" exitCode=0
Sep 09 12:11:59.124863 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.124510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerDied","Data":"528012d1df35cedc83f64efe1c6f3ef741be3221ad0b21469f4782377a1ecb5c"}
Sep 09 12:11:59.502071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.502023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:11:59.502295 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.502148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:59.502295 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:11:59.502176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:11:59.502295 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502201    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:11:59.502445 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502296    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:07.502276613 +0000 UTC m=+50.132864367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:11:59.502445 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502301    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:11:59.502445 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502308    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:11:59.502445 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502353    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:00.502339828 +0000 UTC m=+43.132927581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : configmap "kube-rbac-proxy" not found
Sep 09 12:11:59.502445 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:11:59.502368    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:00.5023622 +0000 UTC m=+43.132949954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:00.106588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.106542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:00.106802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.106643    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:12:00.106802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.106687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:00.106802 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.106732    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:12:00.106802 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.106785    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:12:00.106954 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.106794    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:08.106780974 +0000 UTC m=+50.737368729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:12:00.106954 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.106851    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:08.106832786 +0000 UTC m=+50.737420541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:12:00.106954 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.106866    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:08.106857482 +0000 UTC m=+50.737445235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:12:00.130511 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.130474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" event={"ID":"956af29a-af32-4d5a-8eaa-ce00d3b27996","Type":"ContainerStarted","Data":"fb1dfd4082722191691f62a291a194bc51a41a96a5c65f89dcb262926201fffc"}
Sep 09 12:12:00.156854 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.156802    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d6xx4" podStartSLOduration=8.922014127 podStartE2EDuration="42.156783957s" totalImagesPullingTime="33.23476983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:00.156644084 +0000 UTC m=+42.787231871" watchObservedRunningTime="2026-09-09 12:12:00.156783957 +0000 UTC m=+42.787371732"
Sep 09 12:12:00.509962 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.509922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:00.509962 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.509963    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:00.510175 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.510046    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:12:00.510175 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.510114    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:02.510099531 +0000 UTC m=+45.140687285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : configmap "kube-rbac-proxy" not found
Sep 09 12:12:00.510175 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.510150    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:00.510294 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:00.510223    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:02.510208349 +0000 UTC m=+45.140796120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:00.837973 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:00.837899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gqw7g_4dc63ba9-7b3b-4eaf-9900-4be507b168b7/dns-node-resolver/0.log"
Sep 09 12:12:01.236920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:01.236898    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bm5b2_5b1b1875-e67e-49de-9073-0333de341328/node-ca/0.log"
Sep 09 12:12:02.342886 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.342854    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-247l8"]
Sep 09 12:12:02.372849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.372812    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-247l8"]
Sep 09 12:12:02.373003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.372908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.375459 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.375435    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 12:12:02.375628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.375455    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:12:02.375628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.375455    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 12:12:02.375628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.375532    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 12:12:02.375628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.375462    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fhtqm\""
Sep 09 12:12:02.381037 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.381017    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 12:12:02.437363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.437330    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqnpx_d79601f5-c28a-4c83-8c21-57e7e1d31668/serve-healthcheck-canary/0.log"
Sep 09 12:12:02.524912 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.524879    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-config\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.525057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.524917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-trusted-ca\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.525057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.524945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccrd\" (UniqueName: \"kubernetes.io/projected/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-kube-api-access-mccrd\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.525129 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.525101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:02.525168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.525150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:02.525232 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:02.525217    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:12:02.525293 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:02.525266    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:02.525327 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:02.525317    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:06.525295888 +0000 UTC m=+49.155883641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:02.525367 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:02.525330    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:06.525323102 +0000 UTC m=+49.155910855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : configmap "kube-rbac-proxy" not found
Sep 09 12:12:02.525367 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.525341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-serving-cert\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.626080 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.625994    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-serving-cert\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.626080 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.626062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-config\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.626080 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.626081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-trusted-ca\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.626313 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.626099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccrd\" (UniqueName: \"kubernetes.io/projected/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-kube-api-access-mccrd\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.627025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.626951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-config\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.627399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.627380    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-trusted-ca\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.628754 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.628730    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-serving-cert\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.636833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.636803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccrd\" (UniqueName: \"kubernetes.io/projected/3228e9fa-08dc-4baa-be44-fd7b9be2b7f6-kube-api-access-mccrd\") pod \"console-operator-85b454874b-247l8\" (UID: \"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6\") " pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.682867 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.682694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:02.976409 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:02.976369    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-247l8"]
Sep 09 12:12:02.980024 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:02.979996    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3228e9fa_08dc_4baa_be44_fd7b9be2b7f6.slice/crio-dda128ec55b9153fa1bf4f04369e15201a82ee9b46590d6ce724627269fad02d WatchSource:0}: Error finding container dda128ec55b9153fa1bf4f04369e15201a82ee9b46590d6ce724627269fad02d: Status 404 returned error can't find the container with id dda128ec55b9153fa1bf4f04369e15201a82ee9b46590d6ce724627269fad02d
Sep 09 12:12:03.138920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:03.138885    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-247l8" event={"ID":"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6","Type":"ContainerStarted","Data":"dda128ec55b9153fa1bf4f04369e15201a82ee9b46590d6ce724627269fad02d"}
Sep 09 12:12:06.148631 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.148604    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/0.log"
Sep 09 12:12:06.149102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.148642    2556 generic.go:356] "Generic (PLEG): container finished" podID="3228e9fa-08dc-4baa-be44-fd7b9be2b7f6" containerID="d360b0b8c369ff2b2899fbb1da2d28e8de6cc3b50003522385be880541db6e7e" exitCode=255
Sep 09 12:12:06.149102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.148676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-247l8" event={"ID":"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6","Type":"ContainerDied","Data":"d360b0b8c369ff2b2899fbb1da2d28e8de6cc3b50003522385be880541db6e7e"}
Sep 09 12:12:06.149102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.148988    2556 scope.go:117] "RemoveContainer" containerID="d360b0b8c369ff2b2899fbb1da2d28e8de6cc3b50003522385be880541db6e7e"
Sep 09 12:12:06.561197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.561152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:06.561416 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:06.561215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:06.561416 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:06.561282    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 12:12:06.561416 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:06.561358    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:14.561338334 +0000 UTC m=+57.191926107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : configmap "kube-rbac-proxy" not found
Sep 09 12:12:06.561416 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:06.561379    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:06.561571 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:06.561430    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:14.561414819 +0000 UTC m=+57.192002577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:07.154122 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154093    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:12:07.154537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154452    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/0.log"
Sep 09 12:12:07.154537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154482    2556 generic.go:356] "Generic (PLEG): container finished" podID="3228e9fa-08dc-4baa-be44-fd7b9be2b7f6" containerID="caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920" exitCode=255
Sep 09 12:12:07.154602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154540    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-247l8" event={"ID":"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6","Type":"ContainerDied","Data":"caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920"}
Sep 09 12:12:07.154602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154582    2556 scope.go:117] "RemoveContainer" containerID="d360b0b8c369ff2b2899fbb1da2d28e8de6cc3b50003522385be880541db6e7e"
Sep 09 12:12:07.154803 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.154785    2556 scope.go:117] "RemoveContainer" containerID="caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920"
Sep 09 12:12:07.154980 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:07.154958    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-247l8_openshift-console-operator(3228e9fa-08dc-4baa-be44-fd7b9be2b7f6)\"" pod="openshift-console-operator/console-operator-85b454874b-247l8" podUID="3228e9fa-08dc-4baa-be44-fd7b9be2b7f6"
Sep 09 12:12:07.571520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:07.571479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:07.571721 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:07.571655    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 12:12:07.571769 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:07.571723    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls podName:0878ddec-9390-4909-b2ea-4cb57aea3698 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:23.571706414 +0000 UTC m=+66.202294185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls") pod "dns-default-m5j95" (UID: "0878ddec-9390-4909-b2ea-4cb57aea3698") : secret "dns-default-metrics-tls" not found
Sep 09 12:12:08.159201 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:08.159174    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:12:08.159638 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:08.159513    2556 scope.go:117] "RemoveContainer" containerID="caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920"
Sep 09 12:12:08.159679 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.159657    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-247l8_openshift-console-operator(3228e9fa-08dc-4baa-be44-fd7b9be2b7f6)\"" pod="openshift-console-operator/console-operator-85b454874b-247l8" podUID="3228e9fa-08dc-4baa-be44-fd7b9be2b7f6"
Sep 09 12:12:08.175664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:08.175641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:12:08.175807 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:08.175675    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:08.175807 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:08.175710    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:08.175807 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.175776    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 12:12:08.175807 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.175792    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:24.175775697 +0000 UTC m=+66.806363451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : configmap references non-existent config key: service-ca.crt
Sep 09 12:12:08.176019 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.175827    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls podName:eb1847d7-9261-487a-be2c-972566991417 nodeName:}" failed. No retries permitted until 2026-09-09 12:12:24.175810955 +0000 UTC m=+66.806398710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-hcl8g" (UID: "eb1847d7-9261-487a-be2c-972566991417") : secret "cluster-monitoring-operator-tls" not found
Sep 09 12:12:08.176019 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.175877    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 12:12:08.176019 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:08.175932    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs podName:63cc97ea-6218-4078-b500-bec5db94a85f nodeName:}" failed. No retries permitted until 2026-09-09 12:12:24.175916239 +0000 UTC m=+66.806504006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs") pod "router-default-867b7bd764-5qsj7" (UID: "63cc97ea-6218-4078-b500-bec5db94a85f") : secret "router-metrics-certs-default" not found
Sep 09 12:12:12.683773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:12.683734    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:12.684209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:12.683799    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:12.684209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:12.684122    2556 scope.go:117] "RemoveContainer" containerID="caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920"
Sep 09 12:12:12.684337 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:12.684304    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-247l8_openshift-console-operator(3228e9fa-08dc-4baa-be44-fd7b9be2b7f6)\"" pod="openshift-console-operator/console-operator-85b454874b-247l8" podUID="3228e9fa-08dc-4baa-be44-fd7b9be2b7f6"
Sep 09 12:12:14.631439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:14.631397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:14.631927 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:14.631451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:14.631927 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:14.631542    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:14.631927 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:12:14.631607    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls podName:012a3b9a-7354-42df-b198-2011231deceb nodeName:}" failed. No retries permitted until 2026-09-09 12:12:30.63159299 +0000 UTC m=+73.262180745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mdwth" (UID: "012a3b9a-7354-42df-b198-2011231deceb") : secret "insights-runtime-extractor-tls" not found
Sep 09 12:12:14.632106 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:14.632043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012a3b9a-7354-42df-b198-2011231deceb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:17.082482 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:17.082451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snxqt"
Sep 09 12:12:23.605928 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.605881    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:23.608306 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.608284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0878ddec-9390-4909-b2ea-4cb57aea3698-metrics-tls\") pod \"dns-default-m5j95\" (UID: \"0878ddec-9390-4909-b2ea-4cb57aea3698\") " pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:23.706927 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.706890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:12:23.707112 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.706938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:12:23.709750 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.709730    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 12:12:23.710632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.710615    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 12:12:23.719802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.719779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f66ff2-761b-4503-be4d-8bfb9e55536e-metrics-certs\") pod \"network-metrics-daemon-wctrv\" (UID: \"65f66ff2-761b-4503-be4d-8bfb9e55536e\") " pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:12:23.719902 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.719839    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c195e29-9ded-4d92-bebe-429d6354bd70-original-pull-secret\") pod \"global-pull-secret-syncer-l7hxr\" (UID: \"7c195e29-9ded-4d92-bebe-429d6354bd70\") " pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:12:23.724904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.724884    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s86q\""
Sep 09 12:12:23.732887 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.732872    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wctrv"
Sep 09 12:12:23.786957 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.786910    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mksrl\""
Sep 09 12:12:23.795176 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.795150    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:23.807590 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.807565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:12:23.809938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.809917    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 12:12:23.821208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.821171    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 12:12:23.833081 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.833033    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqwp\" (UniqueName: \"kubernetes.io/projected/3673b08c-2a12-411f-a631-e8c142056089-kube-api-access-mpqwp\") pod \"network-check-target-5gm6c\" (UID: \"3673b08c-2a12-411f-a631-e8c142056089\") " pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:12:23.999348 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:23.999191    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wctrv"]
Sep 09 12:12:24.001865 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.001841    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f66ff2_761b_4503_be4d_8bfb9e55536e.slice/crio-bc95f226d31ef7c6fa06a206c5dc5fcbc1cdf3a84366dea27040c1be59265a2c WatchSource:0}: Error finding container bc95f226d31ef7c6fa06a206c5dc5fcbc1cdf3a84366dea27040c1be59265a2c: Status 404 returned error can't find the container with id bc95f226d31ef7c6fa06a206c5dc5fcbc1cdf3a84366dea27040c1be59265a2c
Sep 09 12:12:24.009311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.009289    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hxr"
Sep 09 12:12:24.019327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.019309    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g5zgt\""
Sep 09 12:12:24.027373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.027357    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:12:24.068154 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.067088    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5j95"]
Sep 09 12:12:24.073973 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.073942    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0878ddec_9390_4909_b2ea_4cb57aea3698.slice/crio-249564bc41df7a20cb761ebd301e9aed3d5c19cda02b35e428096400dba4258c WatchSource:0}: Error finding container 249564bc41df7a20cb761ebd301e9aed3d5c19cda02b35e428096400dba4258c: Status 404 returned error can't find the container with id 249564bc41df7a20cb761ebd301e9aed3d5c19cda02b35e428096400dba4258c
Sep 09 12:12:24.200784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.200750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5j95" event={"ID":"0878ddec-9390-4909-b2ea-4cb57aea3698","Type":"ContainerStarted","Data":"249564bc41df7a20cb761ebd301e9aed3d5c19cda02b35e428096400dba4258c"}
Sep 09 12:12:24.201837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.201812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wctrv" event={"ID":"65f66ff2-761b-4503-be4d-8bfb9e55536e","Type":"ContainerStarted","Data":"bc95f226d31ef7c6fa06a206c5dc5fcbc1cdf3a84366dea27040c1be59265a2c"}
Sep 09 12:12:24.211230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.211206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:12:24.211337 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.211256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:24.211337 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.211283    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:24.211909 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.211885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63cc97ea-6218-4078-b500-bec5db94a85f-service-ca-bundle\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:24.213757 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.213737    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63cc97ea-6218-4078-b500-bec5db94a85f-metrics-certs\") pod \"router-default-867b7bd764-5qsj7\" (UID: \"63cc97ea-6218-4078-b500-bec5db94a85f\") " pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:24.213853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.213744    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb1847d7-9261-487a-be2c-972566991417-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-hcl8g\" (UID: \"eb1847d7-9261-487a-be2c-972566991417\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:12:24.280074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.280004    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l7hxr"]
Sep 09 12:12:24.282978 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.282952    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c195e29_9ded_4d92_bebe_429d6354bd70.slice/crio-8e2692863d164a575818397d8cbf88763202a6d761232c930db66eecb52471fc WatchSource:0}: Error finding container 8e2692863d164a575818397d8cbf88763202a6d761232c930db66eecb52471fc: Status 404 returned error can't find the container with id 8e2692863d164a575818397d8cbf88763202a6d761232c930db66eecb52471fc
Sep 09 12:12:24.304755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.304721    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gm6c"]
Sep 09 12:12:24.308395 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.308371    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3673b08c_2a12_411f_a631_e8c142056089.slice/crio-f42d919bfa3fcb35ab5e1095635a647e2598ec9c227cb37af3bb58ced201c7c2 WatchSource:0}: Error finding container f42d919bfa3fcb35ab5e1095635a647e2598ec9c227cb37af3bb58ced201c7c2: Status 404 returned error can't find the container with id f42d919bfa3fcb35ab5e1095635a647e2598ec9c227cb37af3bb58ced201c7c2
Sep 09 12:12:24.411769 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.411705    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x5mnw\""
Sep 09 12:12:24.419847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.419803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"
Sep 09 12:12:24.426975 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.426949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-959qs\""
Sep 09 12:12:24.435215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.435032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:24.703355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.703117    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g"]
Sep 09 12:12:24.727355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:24.727323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-867b7bd764-5qsj7"]
Sep 09 12:12:24.871630 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.871586    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cc97ea_6218_4078_b500_bec5db94a85f.slice/crio-b8b5a05a671e0ccd212e1e05ba5321318514ca2fdc8d730d9af6a49073d75c2f WatchSource:0}: Error finding container b8b5a05a671e0ccd212e1e05ba5321318514ca2fdc8d730d9af6a49073d75c2f: Status 404 returned error can't find the container with id b8b5a05a671e0ccd212e1e05ba5321318514ca2fdc8d730d9af6a49073d75c2f
Sep 09 12:12:24.872486 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:24.872196    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1847d7_9261_487a_be2c_972566991417.slice/crio-a0e24cea7cb23694c58ad1dcd2e62b2781eaca08e01daa95053a27725a477757 WatchSource:0}: Error finding container a0e24cea7cb23694c58ad1dcd2e62b2781eaca08e01daa95053a27725a477757: Status 404 returned error can't find the container with id a0e24cea7cb23694c58ad1dcd2e62b2781eaca08e01daa95053a27725a477757
Sep 09 12:12:25.207047 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.206967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g" event={"ID":"eb1847d7-9261-487a-be2c-972566991417","Type":"ContainerStarted","Data":"a0e24cea7cb23694c58ad1dcd2e62b2781eaca08e01daa95053a27725a477757"}
Sep 09 12:12:25.208337 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.208313    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gm6c" event={"ID":"3673b08c-2a12-411f-a631-e8c142056089","Type":"ContainerStarted","Data":"f42d919bfa3fcb35ab5e1095635a647e2598ec9c227cb37af3bb58ced201c7c2"}
Sep 09 12:12:25.209606 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.209582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l7hxr" event={"ID":"7c195e29-9ded-4d92-bebe-429d6354bd70","Type":"ContainerStarted","Data":"8e2692863d164a575818397d8cbf88763202a6d761232c930db66eecb52471fc"}
Sep 09 12:12:25.211566 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.211539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wctrv" event={"ID":"65f66ff2-761b-4503-be4d-8bfb9e55536e","Type":"ContainerStarted","Data":"0b16b7ce7cb1dcad80397794ec6da04c659ee7441433c50f5d9086a9e9de23f8"}
Sep 09 12:12:25.211656 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.211584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wctrv" event={"ID":"65f66ff2-761b-4503-be4d-8bfb9e55536e","Type":"ContainerStarted","Data":"4d6938bfcd4854ecf3c68e4343d1fb03f4644cdc304fb6f082a26d6b9ce9f18d"}
Sep 09 12:12:25.213209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.213190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-867b7bd764-5qsj7" event={"ID":"63cc97ea-6218-4078-b500-bec5db94a85f","Type":"ContainerStarted","Data":"58c6b4917d8d960f3659d17b0d6456c50a3a36abbfa20c1cac23bfd800a77315"}
Sep 09 12:12:25.213330 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.213221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-867b7bd764-5qsj7" event={"ID":"63cc97ea-6218-4078-b500-bec5db94a85f","Type":"ContainerStarted","Data":"b8b5a05a671e0ccd212e1e05ba5321318514ca2fdc8d730d9af6a49073d75c2f"}
Sep 09 12:12:25.230948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.230884    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wctrv" podStartSLOduration=66.335739457 podStartE2EDuration="1m7.230864566s" totalImagesPullingTime="895.125109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:25.230578626 +0000 UTC m=+67.861166402" watchObservedRunningTime="2026-09-09 12:12:25.230864566 +0000 UTC m=+67.861452344"
Sep 09 12:12:25.250628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.250558    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-867b7bd764-5qsj7" podStartSLOduration=33.250534542 podStartE2EDuration="33.250534542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:25.250195622 +0000 UTC m=+67.880783400" watchObservedRunningTime="2026-09-09 12:12:25.250534542 +0000 UTC m=+67.881122320"
Sep 09 12:12:25.435779 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.435745    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:25.438549 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.438514    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:25.897745 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:25.897714    2556 scope.go:117] "RemoveContainer" containerID="caa3d00d9d92c61caa4f7e075f3c887796b2500180e3107e026476eaeac6a920"
Sep 09 12:12:26.221293 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.221266    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:12:26.221463 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.221396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-247l8" event={"ID":"3228e9fa-08dc-4baa-be44-fd7b9be2b7f6","Type":"ContainerStarted","Data":"eba844fe76bef1c0292906e89bb3d16ba0965ef43aa0be57acdcf04ca5ad3cd0"}
Sep 09 12:12:26.223726 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.223701    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:26.223864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.223769    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:26.223864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.223835    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-867b7bd764-5qsj7"
Sep 09 12:12:26.265710 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.265640    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-247l8" podStartSLOduration=21.672700795 podStartE2EDuration="24.265614841s" totalImagesPullingTime="2.592914046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:26.244559283 +0000 UTC m=+68.875147060" watchObservedRunningTime="2026-09-09 12:12:26.265614841 +0000 UTC m=+68.896202618"
Sep 09 12:12:26.480110 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:26.480029    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-247l8"
Sep 09 12:12:27.228161 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:27.228116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g" event={"ID":"eb1847d7-9261-487a-be2c-972566991417","Type":"ContainerStarted","Data":"125df1bf10c7cdaca0d0db8e727edf95b9ec7fb9d74750344eb461b51c7b3aa3"}
Sep 09 12:12:30.261095 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.261007    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-hcl8g" podStartSLOduration=36.052105342 podStartE2EDuration="38.260984301s" totalImagesPullingTime="2.208878959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:27.269486622 +0000 UTC m=+69.900074399" watchObservedRunningTime="2026-09-09 12:12:30.260984301 +0000 UTC m=+72.891572079"
Sep 09 12:12:30.262713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.262685    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"]
Sep 09 12:12:30.270149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.270121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.276091 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.276067    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 12:12:30.276297 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.276073    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 12:12:30.276297 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.276260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 12:12:30.277025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.277003    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 12:12:30.277261 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.277220    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 12:12:30.277470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.277455    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 12:12:30.277804 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.277657    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 12:12:30.304677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.304335    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"]
Sep 09 12:12:30.367281 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.367449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97s2q\" (UniqueName: \"kubernetes.io/projected/dbbe3d23-0d83-4228-9b65-27404a6b92ee-kube-api-access-97s2q\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.367449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367353    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.367449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367414    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.367449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367437    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.367672 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.367472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbe3d23-0d83-4228-9b65-27404a6b92ee-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.467953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.467919    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.467953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.467954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.468210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.467977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbe3d23-0d83-4228-9b65-27404a6b92ee-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.468210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.468007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.468210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.468024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97s2q\" (UniqueName: \"kubernetes.io/projected/dbbe3d23-0d83-4228-9b65-27404a6b92ee-kube-api-access-97s2q\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.468210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.468175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.468882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.468834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbe3d23-0d83-4228-9b65-27404a6b92ee-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.470569 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.470545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-ca\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.470723 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.470704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.471019 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.470998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.471058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.471022    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbbe3d23-0d83-4228-9b65-27404a6b92ee-hub\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.498204 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.498168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97s2q\" (UniqueName: \"kubernetes.io/projected/dbbe3d23-0d83-4228-9b65-27404a6b92ee-kube-api-access-97s2q\") pod \"cluster-proxy-proxy-agent-5c47599568-7jlgx\" (UID: \"dbbe3d23-0d83-4228-9b65-27404a6b92ee\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.591495 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.591409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"
Sep 09 12:12:30.669877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.669845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:30.672232 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.672213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012a3b9a-7354-42df-b198-2011231deceb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdwth\" (UID: \"012a3b9a-7354-42df-b198-2011231deceb\") " pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:30.817936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.817900    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lrqz2\""
Sep 09 12:12:30.826013 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:30.825987    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mdwth"
Sep 09 12:12:31.196919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.196883    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mdwth"]
Sep 09 12:12:31.203816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.203792    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx"]
Sep 09 12:12:31.206797 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:31.206767    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbe3d23_0d83_4228_9b65_27404a6b92ee.slice/crio-b762cdbcf066c80c4f433ea3c5176f0f3dc280bce32bf862afd402c1c7af86ce WatchSource:0}: Error finding container b762cdbcf066c80c4f433ea3c5176f0f3dc280bce32bf862afd402c1c7af86ce: Status 404 returned error can't find the container with id b762cdbcf066c80c4f433ea3c5176f0f3dc280bce32bf862afd402c1c7af86ce
Sep 09 12:12:31.245212 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.245180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5j95" event={"ID":"0878ddec-9390-4909-b2ea-4cb57aea3698","Type":"ContainerStarted","Data":"21bd3c25048b6ce8dbf96c72493c390fcd0d06a8d68df77e89c369978c9bcd68"}
Sep 09 12:12:31.246570 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.246541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx" event={"ID":"dbbe3d23-0d83-4228-9b65-27404a6b92ee","Type":"ContainerStarted","Data":"b762cdbcf066c80c4f433ea3c5176f0f3dc280bce32bf862afd402c1c7af86ce"}
Sep 09 12:12:31.248265 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.248191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gm6c" event={"ID":"3673b08c-2a12-411f-a631-e8c142056089","Type":"ContainerStarted","Data":"0da6f8ab48984596399ea629af467bc7bb1a69eef69c643437dde53fa3ed866a"}
Sep 09 12:12:31.248368 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.248304    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:12:31.249868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.249842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l7hxr" event={"ID":"7c195e29-9ded-4d92-bebe-429d6354bd70","Type":"ContainerStarted","Data":"c6ded739871dc32928d38bfc23606be32b1923bc11d4d34b2ed5994ffab871e3"}
Sep 09 12:12:31.251124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.251104    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdwth" event={"ID":"012a3b9a-7354-42df-b198-2011231deceb","Type":"ContainerStarted","Data":"dfbec6e3966cd0637fbb29514e55bcc1e459dc70039a9dc5d100f2fc265195a5"}
Sep 09 12:12:31.268885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.268829    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gm6c" podStartSLOduration=66.669573084 podStartE2EDuration="1m13.268808328s" totalImagesPullingTime="6.599235244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:31.267839237 +0000 UTC m=+73.898427014" watchObservedRunningTime="2026-09-09 12:12:31.268808328 +0000 UTC m=+73.899396105"
Sep 09 12:12:31.286665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:31.286592    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l7hxr" podStartSLOduration=66.648162879 podStartE2EDuration="1m13.286574224s" totalImagesPullingTime="6.638411345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:31.285589311 +0000 UTC m=+73.916177100" watchObservedRunningTime="2026-09-09 12:12:31.286574224 +0000 UTC m=+73.917162009"
Sep 09 12:12:32.257267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:32.257211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdwth" event={"ID":"012a3b9a-7354-42df-b198-2011231deceb","Type":"ContainerStarted","Data":"9a14e6306aeb43e0f27b3f73c5185cf7832cad0b084e60e4f4e8c3633c9be6a7"}
Sep 09 12:12:32.259833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:32.259354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5j95" event={"ID":"0878ddec-9390-4909-b2ea-4cb57aea3698","Type":"ContainerStarted","Data":"2d459511a6874cab792ad9f1121d7a6841a6512097e3f178b849fc8409d15e05"}
Sep 09 12:12:32.284346 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:32.284289    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m5j95" podStartSLOduration=34.450957094 podStartE2EDuration="41.284265419s" totalImagesPullingTime="6.833308325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:32.283266449 +0000 UTC m=+74.913854223" watchObservedRunningTime="2026-09-09 12:12:32.284265419 +0000 UTC m=+74.914853195"
Sep 09 12:12:33.265767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:33.265709    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdwth" event={"ID":"012a3b9a-7354-42df-b198-2011231deceb","Type":"ContainerStarted","Data":"f342177df1061719f3b16edbb0c2b9c59d43084cb8329e6ab00f6ce69e8d242e"}
Sep 09 12:12:33.265986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:33.265972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:36.279717 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.279678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx" event={"ID":"dbbe3d23-0d83-4228-9b65-27404a6b92ee","Type":"ContainerStarted","Data":"42c84a3f0eb9b807b8512f4a55c215a50aa6ce3d89cdad5e4ce6b72aeaa7ea1f"}
Sep 09 12:12:36.280192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.279730    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx" event={"ID":"dbbe3d23-0d83-4228-9b65-27404a6b92ee","Type":"ContainerStarted","Data":"8ee8351c42c233aa714dd8842bf3beb6debaf52678c07819d0cbb3b65a8dd185"}
Sep 09 12:12:36.280192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.279746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx" event={"ID":"dbbe3d23-0d83-4228-9b65-27404a6b92ee","Type":"ContainerStarted","Data":"1e8b70dfdf341e9ec4dc0997465e2fb88f7762ce8fb2dbf744cd3b98f8d502dd"}
Sep 09 12:12:36.281834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.281811    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdwth" event={"ID":"012a3b9a-7354-42df-b198-2011231deceb","Type":"ContainerStarted","Data":"676539fbb0d42eb78e0cfaf31a1fc4fdb88622e8c2c65747365c5a381a39d862"}
Sep 09 12:12:36.301179 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.301129    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c47599568-7jlgx" podStartSLOduration=1.907935911 podStartE2EDuration="6.301115878s" totalImagesPullingTime="4.393179967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:36.299872818 +0000 UTC m=+78.930460593" watchObservedRunningTime="2026-09-09 12:12:36.301115878 +0000 UTC m=+78.931703656"
Sep 09 12:12:36.320291 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:36.320211    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mdwth" podStartSLOduration=34.169370314 podStartE2EDuration="38.320193553s" totalImagesPullingTime="4.150823239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:11:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:36.318977614 +0000 UTC m=+78.949565392" watchObservedRunningTime="2026-09-09 12:12:36.320193553 +0000 UTC m=+78.950781329"
Sep 09 12:12:43.128653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.128539    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-krcpp"]
Sep 09 12:12:43.133385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.133355    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.136192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.136170    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 12:12:43.136339 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.136172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 12:12:43.136593 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.136571    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwtql\""
Sep 09 12:12:43.137182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.137155    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 12:12:43.137311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.137192    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 12:12:43.266312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-tls\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266312    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-metrics-client-ca\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-wtmp\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266485    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbf8\" (UniqueName: \"kubernetes.io/projected/81e77ee6-53ac-462b-a314-315459b3e9a1-kube-api-access-vhbf8\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-root\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266539    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-textfile\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.266578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.266564    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-sys\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.272027 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.272006    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m5j95"
Sep 09 12:12:43.366979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.366947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-root\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.366985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-textfile\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-sys\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-root\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-tls\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-metrics-client-ca\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-sys\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-wtmp\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbf8\" (UniqueName: \"kubernetes.io/projected/81e77ee6-53ac-462b-a314-315459b3e9a1-kube-api-access-vhbf8\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367458    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-wtmp\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.367729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-textfile\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.368002 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.367978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-metrics-client-ca\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.368273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.368227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.369562 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.369540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.369644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.369628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e77ee6-53ac-462b-a314-315459b3e9a1-node-exporter-tls\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.375873 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.375852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbf8\" (UniqueName: \"kubernetes.io/projected/81e77ee6-53ac-462b-a314-315459b3e9a1-kube-api-access-vhbf8\") pod \"node-exporter-krcpp\" (UID: \"81e77ee6-53ac-462b-a314-315459b3e9a1\") " pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.443222 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:43.443138    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-krcpp"
Sep 09 12:12:43.454129 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:43.454102    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e77ee6_53ac_462b_a314_315459b3e9a1.slice/crio-17880ecb4ca6e82d1f69a60123fb29a39dfd88e5ad697699eec6edc62d841ead WatchSource:0}: Error finding container 17880ecb4ca6e82d1f69a60123fb29a39dfd88e5ad697699eec6edc62d841ead: Status 404 returned error can't find the container with id 17880ecb4ca6e82d1f69a60123fb29a39dfd88e5ad697699eec6edc62d841ead
Sep 09 12:12:44.317279 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:44.317220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-krcpp" event={"ID":"81e77ee6-53ac-462b-a314-315459b3e9a1","Type":"ContainerStarted","Data":"17880ecb4ca6e82d1f69a60123fb29a39dfd88e5ad697699eec6edc62d841ead"}
Sep 09 12:12:45.323420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:45.323391    2556 generic.go:356] "Generic (PLEG): container finished" podID="81e77ee6-53ac-462b-a314-315459b3e9a1" containerID="225b70dbbec58c53db441c3ee0c4d38dfd62e64fe96490182d12094af46d9f69" exitCode=0
Sep 09 12:12:45.323828 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:45.323440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-krcpp" event={"ID":"81e77ee6-53ac-462b-a314-315459b3e9a1","Type":"ContainerDied","Data":"225b70dbbec58c53db441c3ee0c4d38dfd62e64fe96490182d12094af46d9f69"}
Sep 09 12:12:46.329826 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:46.329788    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-krcpp" event={"ID":"81e77ee6-53ac-462b-a314-315459b3e9a1","Type":"ContainerStarted","Data":"684b8932eaa58308aa1392e9c58654ab5b52a9613345523ef57d9eb4cd73e57d"}
Sep 09 12:12:46.330210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:46.329853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-krcpp" event={"ID":"81e77ee6-53ac-462b-a314-315459b3e9a1","Type":"ContainerStarted","Data":"8fabfa092b3ddf3c61594a88fe495e46f94168cdd1199b74e1f569f08e3b62bb"}
Sep 09 12:12:46.352554 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:46.352501    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-krcpp" podStartSLOduration=2.244349511 podStartE2EDuration="3.352484461s" totalImagesPullingTime="1.039088911s" totalInitContainerRuntime="69.046039ms" isStatefulPod=true podCreationTimestamp="2026-09-09 12:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:46.351452012 +0000 UTC m=+88.982039789" watchObservedRunningTime="2026-09-09 12:12:46.352484461 +0000 UTC m=+88.983072238"
Sep 09 12:12:54.830410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.830374    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:12:54.834589 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.834573    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.842262 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.842227    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 12:12:54.843517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.843501    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 12:12:54.843723 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.843710    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 12:12:54.845862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.845842    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 12:12:54.845973 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.845843    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 12:12:54.846127 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.846114    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 12:12:54.846961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.846947    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 12:12:54.849934 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.849914    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:12:54.875568 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.875538    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 12:12:54.884497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.884478    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pm6s\""
Sep 09 12:12:54.948156 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcdm\" (UniqueName: \"kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:54.948331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:54.948309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049005 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.048969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcdm\" (UniqueName: \"kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049366 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049366 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049843 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.049959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.049817    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.050360 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.050338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.051988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.051965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.052068 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.051988    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.058827 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.058805    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcdm\" (UniqueName: \"kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm\") pod \"console-58fc855dcd-s276r\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") " pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.143561 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.143468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:12:55.419364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:55.419294    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:12:55.422269 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:12:55.422225    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae9b624_ae94_45fd_99e3_453196b2f7be.slice/crio-0e310982379f67dedb6542ee932645c8dbdc545f89b188a478861453e918f1df WatchSource:0}: Error finding container 0e310982379f67dedb6542ee932645c8dbdc545f89b188a478861453e918f1df: Status 404 returned error can't find the container with id 0e310982379f67dedb6542ee932645c8dbdc545f89b188a478861453e918f1df
Sep 09 12:12:56.366781 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:56.366746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58fc855dcd-s276r" event={"ID":"eae9b624-ae94-45fd-99e3-453196b2f7be","Type":"ContainerStarted","Data":"0e310982379f67dedb6542ee932645c8dbdc545f89b188a478861453e918f1df"}
Sep 09 12:12:58.376157 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:58.376118    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58fc855dcd-s276r" event={"ID":"eae9b624-ae94-45fd-99e3-453196b2f7be","Type":"ContainerStarted","Data":"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"}
Sep 09 12:12:58.395637 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:12:58.395569    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58fc855dcd-s276r" podStartSLOduration=1.578751027 podStartE2EDuration="4.39555373s" totalImagesPullingTime="2.816802703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:12:58.394953652 +0000 UTC m=+101.025541431" watchObservedRunningTime="2026-09-09 12:12:58.39555373 +0000 UTC m=+101.026141506"
Sep 09 12:13:02.262234 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:02.262205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gm6c"
Sep 09 12:13:03.395637 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:03.395605    2556 generic.go:356] "Generic (PLEG): container finished" podID="640e5fd6-5c1f-4b28-a14d-4cb59e947583" containerID="9ba37ac91d37e3c0875e962d166d176a630a8f3eb94d46cf575899b20d154cab" exitCode=0
Sep 09 12:13:03.396014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:03.395682    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc" event={"ID":"640e5fd6-5c1f-4b28-a14d-4cb59e947583","Type":"ContainerDied","Data":"9ba37ac91d37e3c0875e962d166d176a630a8f3eb94d46cf575899b20d154cab"}
Sep 09 12:13:03.396079 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:03.396018    2556 scope.go:117] "RemoveContainer" containerID="9ba37ac91d37e3c0875e962d166d176a630a8f3eb94d46cf575899b20d154cab"
Sep 09 12:13:04.401009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:04.400978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-75fzc" event={"ID":"640e5fd6-5c1f-4b28-a14d-4cb59e947583","Type":"ContainerStarted","Data":"4ac39da0d1ff794af12d43e46e2c35f70f20971aff6fa2f5a6e630c8252963ce"}
Sep 09 12:13:05.143604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:05.143566    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:13:05.143799 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:05.143657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:13:05.148399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:05.148379    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:13:05.409051 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:05.408983    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:13:13.436041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:13.436004    2556 generic.go:356] "Generic (PLEG): container finished" podID="0ef14022-b373-4895-8eff-b16e98fa7029" containerID="beed1bca0cfd04c6c83e558a6d31f2c9921ef8c047cb670d4703fb4ff383a8be" exitCode=0
Sep 09 12:13:13.436569 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:13.436085    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7" event={"ID":"0ef14022-b373-4895-8eff-b16e98fa7029","Type":"ContainerDied","Data":"beed1bca0cfd04c6c83e558a6d31f2c9921ef8c047cb670d4703fb4ff383a8be"}
Sep 09 12:13:13.436569 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:13.436515    2556 scope.go:117] "RemoveContainer" containerID="beed1bca0cfd04c6c83e558a6d31f2c9921ef8c047cb670d4703fb4ff383a8be"
Sep 09 12:13:14.441615 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:13:14.441585    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xwht7" event={"ID":"0ef14022-b373-4895-8eff-b16e98fa7029","Type":"ContainerStarted","Data":"5abd407db8209791d16a7bd8d765b55f426ca661ce3e7ed235f4d9f8a0b185e1"}
Sep 09 12:14:10.901476 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:10.901397    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:14:10.904504 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:10.904484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:10.920180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:10.920150    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:14:11.037790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.037754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.037790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.037788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ftx\" (UniqueName: \"kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.038000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.037820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.038000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.037917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.038000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.037974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.038116 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.038083    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.038157 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.038112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.138847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.138847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138872    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.138991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ftx\" (UniqueName: \"kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.139018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139735 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.139674    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.139735 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.139731    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.140087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.139898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.140087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.139932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.141519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.141489    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.141694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.141676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.148786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.148758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ftx\" (UniqueName: \"kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx\") pod \"console-8d765bd-gpsz4\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") " pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.214771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.214734    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:11.491706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.491629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:14:11.494940 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:14:11.494910    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5547e8c7_d5cb_436f_990e_e10826ffe033.slice/crio-321144f080be50e69e7bc151d8c3d05a8646dae75e0069f09ca7dedad9441ab0 WatchSource:0}: Error finding container 321144f080be50e69e7bc151d8c3d05a8646dae75e0069f09ca7dedad9441ab0: Status 404 returned error can't find the container with id 321144f080be50e69e7bc151d8c3d05a8646dae75e0069f09ca7dedad9441ab0
Sep 09 12:14:11.655332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.655300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d765bd-gpsz4" event={"ID":"5547e8c7-d5cb-436f-990e-e10826ffe033","Type":"ContainerStarted","Data":"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"}
Sep 09 12:14:11.655536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.655356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d765bd-gpsz4" event={"ID":"5547e8c7-d5cb-436f-990e-e10826ffe033","Type":"ContainerStarted","Data":"321144f080be50e69e7bc151d8c3d05a8646dae75e0069f09ca7dedad9441ab0"}
Sep 09 12:14:11.675015 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:11.674948    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d765bd-gpsz4" podStartSLOduration=1.674927225 podStartE2EDuration="1.674927225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:14:11.67459306 +0000 UTC m=+174.305180837" watchObservedRunningTime="2026-09-09 12:14:11.674927225 +0000 UTC m=+174.305515003"
Sep 09 12:14:21.215755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:21.215696    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:21.215755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:21.215768    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:21.220744 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:21.220720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:21.694854 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:21.694770    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:14:21.742918 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:21.742877    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:14:46.770169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:46.770106    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58fc855dcd-s276r" podUID="eae9b624-ae94-45fd-99e3-453196b2f7be" containerName="console" containerID="cri-o://2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9" gracePeriod=15
Sep 09 12:14:47.017624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.017599    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:14:47.107790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107690    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.107790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107740    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.107790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107767    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.107790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107789    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mcdm\" (UniqueName: \"kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.108103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107833    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.108103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107871    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.108103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.107918    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config\") pod \"eae9b624-ae94-45fd-99e3-453196b2f7be\" (UID: \"eae9b624-ae94-45fd-99e3-453196b2f7be\") "
Sep 09 12:14:47.108320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.108286    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca" (OuterVolumeSpecName: "service-ca") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:14:47.108320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.108298    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:14:47.108480 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.108451    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:14:47.108539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.108492    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config" (OuterVolumeSpecName: "console-config") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:14:47.110184 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.110164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:14:47.110267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.110226    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:14:47.110311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.110266    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm" (OuterVolumeSpecName: "kube-api-access-4mcdm") pod "eae9b624-ae94-45fd-99e3-453196b2f7be" (UID: "eae9b624-ae94-45fd-99e3-453196b2f7be"). InnerVolumeSpecName "kube-api-access-4mcdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:14:47.208881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208842    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-console-config\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.208881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208880    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-oauth-config\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.209100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208896    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-oauth-serving-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.209100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208908    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae9b624-ae94-45fd-99e3-453196b2f7be-console-serving-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.209100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208919    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcdm\" (UniqueName: \"kubernetes.io/projected/eae9b624-ae94-45fd-99e3-453196b2f7be-kube-api-access-4mcdm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.209100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208938    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.209100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.208949    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae9b624-ae94-45fd-99e3-453196b2f7be-trusted-ca-bundle\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:14:47.784490 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.784452    2556 generic.go:356] "Generic (PLEG): container finished" podID="eae9b624-ae94-45fd-99e3-453196b2f7be" containerID="2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9" exitCode=0
Sep 09 12:14:47.784874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.784499    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58fc855dcd-s276r"
Sep 09 12:14:47.784874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.784540    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58fc855dcd-s276r" event={"ID":"eae9b624-ae94-45fd-99e3-453196b2f7be","Type":"ContainerDied","Data":"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"}
Sep 09 12:14:47.784874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.784594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58fc855dcd-s276r" event={"ID":"eae9b624-ae94-45fd-99e3-453196b2f7be","Type":"ContainerDied","Data":"0e310982379f67dedb6542ee932645c8dbdc545f89b188a478861453e918f1df"}
Sep 09 12:14:47.784874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.784615    2556 scope.go:117] "RemoveContainer" containerID="2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"
Sep 09 12:14:47.793389 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.793371    2556 scope.go:117] "RemoveContainer" containerID="2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"
Sep 09 12:14:47.793675 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:14:47.793654    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9\": container with ID starting with 2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9 not found: ID does not exist" containerID="2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"
Sep 09 12:14:47.793724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.793684    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9"} err="failed to get container status \"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9\": rpc error: code = NotFound desc = could not find container \"2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9\": container with ID starting with 2dcc2cb68eeca203f4b94867cac6bcac6eae067374c0cf2f4c83082b8aa87cf9 not found: ID does not exist"
Sep 09 12:14:47.813634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.813603    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:14:47.816102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.816076    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58fc855dcd-s276r"]
Sep 09 12:14:47.899992 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:14:47.899957    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae9b624-ae94-45fd-99e3-453196b2f7be" path="/var/lib/kubelet/pods/eae9b624-ae94-45fd-99e3-453196b2f7be/volumes"
Sep 09 12:15:37.551256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:15:37.551204    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:16:02.573078 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.573038    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8d765bd-gpsz4" podUID="5547e8c7-d5cb-436f-990e-e10826ffe033" containerName="console" containerID="cri-o://f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2" gracePeriod=15
Sep 09 12:16:02.825336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.825227    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:16:02.867561 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867532    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867572    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867597    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867612    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867629    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78ftx\" (UniqueName: \"kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.867982 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867746    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert\") pod \"5547e8c7-d5cb-436f-990e-e10826ffe033\" (UID: \"5547e8c7-d5cb-436f-990e-e10826ffe033\") "
Sep 09 12:16:02.868069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.867962    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config" (OuterVolumeSpecName: "console-config") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:16:02.868132 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.868030    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca" (OuterVolumeSpecName: "service-ca") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:16:02.868181 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.868148    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:16:02.868278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.868228    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:16:02.869938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.869913    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:16:02.870039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.869939    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:16:02.870039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.869960    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx" (OuterVolumeSpecName: "kube-api-access-78ftx") pod "5547e8c7-d5cb-436f-990e-e10826ffe033" (UID: "5547e8c7-d5cb-436f-990e-e10826ffe033"). InnerVolumeSpecName "kube-api-access-78ftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:16:02.969629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969591    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ftx\" (UniqueName: \"kubernetes.io/projected/5547e8c7-d5cb-436f-990e-e10826ffe033-kube-api-access-78ftx\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969622    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-oauth-serving-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969632    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-console-config\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969642    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-serving-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969651    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969659    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5547e8c7-d5cb-436f-990e-e10826ffe033-console-oauth-config\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:02.969862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:02.969667    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5547e8c7-d5cb-436f-990e-e10826ffe033-trusted-ca-bundle\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:16:03.072996 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.072961    2556 generic.go:356] "Generic (PLEG): container finished" podID="5547e8c7-d5cb-436f-990e-e10826ffe033" containerID="f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2" exitCode=0
Sep 09 12:16:03.073137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.073011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d765bd-gpsz4" event={"ID":"5547e8c7-d5cb-436f-990e-e10826ffe033","Type":"ContainerDied","Data":"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"}
Sep 09 12:16:03.073137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.073015    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d765bd-gpsz4"
Sep 09 12:16:03.073137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.073079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d765bd-gpsz4" event={"ID":"5547e8c7-d5cb-436f-990e-e10826ffe033","Type":"ContainerDied","Data":"321144f080be50e69e7bc151d8c3d05a8646dae75e0069f09ca7dedad9441ab0"}
Sep 09 12:16:03.073137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.073102    2556 scope.go:117] "RemoveContainer" containerID="f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"
Sep 09 12:16:03.083320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.083199    2556 scope.go:117] "RemoveContainer" containerID="f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"
Sep 09 12:16:03.083685 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:16:03.083652    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2\": container with ID starting with f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2 not found: ID does not exist" containerID="f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"
Sep 09 12:16:03.083778 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.083696    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2"} err="failed to get container status \"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2\": rpc error: code = NotFound desc = could not find container \"f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2\": container with ID starting with f8a5776a96b016e53b324be3a7398aa1703712a29b5726394f365a6836740ee2 not found: ID does not exist"
Sep 09 12:16:03.098134 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.098108    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:16:03.104575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.104550    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d765bd-gpsz4"]
Sep 09 12:16:03.900413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:03.900379    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5547e8c7-d5cb-436f-990e-e10826ffe033" path="/var/lib/kubelet/pods/5547e8c7-d5cb-436f-990e-e10826ffe033/volumes"
Sep 09 12:16:17.764311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:17.764283    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:16:17.765704 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:17.765684    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:16:17.768278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:17.768258    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:16:17.769819 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:17.769803    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:16:17.775791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:16:17.775771    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 12:17:11.364997 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.364912    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9p9cv"]
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365153    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5547e8c7-d5cb-436f-990e-e10826ffe033" containerName="console"
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365163    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5547e8c7-d5cb-436f-990e-e10826ffe033" containerName="console"
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365173    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae9b624-ae94-45fd-99e3-453196b2f7be" containerName="console"
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365178    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae9b624-ae94-45fd-99e3-453196b2f7be" containerName="console"
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365217    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae9b624-ae94-45fd-99e3-453196b2f7be" containerName="console"
Sep 09 12:17:11.365413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.365224    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5547e8c7-d5cb-436f-990e-e10826ffe033" containerName="console"
Sep 09 12:17:11.367969 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.367953    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.370925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.370902    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 12:17:11.371194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.371176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7hl6f\""
Sep 09 12:17:11.371870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.371855    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 12:17:11.378896 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.378873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9p9cv"]
Sep 09 12:17:11.479558 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.479518    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgq4\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-kube-api-access-bmgq4\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.479739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.479567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.580894 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.580849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.581029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.580941    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgq4\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-kube-api-access-bmgq4\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.590784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.590755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.590898 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.590788    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgq4\" (UniqueName: \"kubernetes.io/projected/a8eea6de-a028-4370-9321-6a5caa69308a-kube-api-access-bmgq4\") pod \"cert-manager-webhook-6486ffd997-9p9cv\" (UID: \"a8eea6de-a028-4370-9321-6a5caa69308a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.676886 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.676807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:11.948550 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.948489    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9p9cv"]
Sep 09 12:17:11.951054 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:17:11.951028    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8eea6de_a028_4370_9321_6a5caa69308a.slice/crio-7434f0206b985afba2625350fac9110075aacb6b68f74e50960c45f165c1a3c2 WatchSource:0}: Error finding container 7434f0206b985afba2625350fac9110075aacb6b68f74e50960c45f165c1a3c2: Status 404 returned error can't find the container with id 7434f0206b985afba2625350fac9110075aacb6b68f74e50960c45f165c1a3c2
Sep 09 12:17:11.953094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:11.953077    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:17:12.314261 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:12.314210    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv" event={"ID":"a8eea6de-a028-4370-9321-6a5caa69308a","Type":"ContainerStarted","Data":"7434f0206b985afba2625350fac9110075aacb6b68f74e50960c45f165c1a3c2"}
Sep 09 12:17:15.326627 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:15.326580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv" event={"ID":"a8eea6de-a028-4370-9321-6a5caa69308a","Type":"ContainerStarted","Data":"a9c34118b7efba3c81b9053148f6bd317a5b2fe8ce910fbb5d69370ba52fa34e"}
Sep 09 12:17:15.327062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:15.326680    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:15.346636 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:15.346585    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv" podStartSLOduration=1.1987400830000001 podStartE2EDuration="4.34656872s" totalImagesPullingTime="3.147828637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:17:15.345361356 +0000 UTC m=+357.975949133" watchObservedRunningTime="2026-09-09 12:17:15.34656872 +0000 UTC m=+357.977156495"
Sep 09 12:17:21.331504 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:21.331473    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9p9cv"
Sep 09 12:17:38.596673 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.596637    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55c5f847b8-pfdfg"]
Sep 09 12:17:38.601748 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.601725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.618279 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.618215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 12:17:38.619144 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.619126    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 12:17:38.619267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.619211    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 12:17:38.619267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.619223    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 12:17:38.619390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.619220    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 12:17:38.619446 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.619215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 12:17:38.622773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.622756    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 12:17:38.627889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.627863    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c5f847b8-pfdfg"]
Sep 09 12:17:38.630402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.630375    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pm6s\""
Sep 09 12:17:38.651977 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.651942    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 12:17:38.681039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-oauth-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-trusted-ca-bundle\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-service-ca\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-console-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681136    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqm6\" (UniqueName: \"kubernetes.io/projected/134d2744-9502-4faa-8c43-db5565cda735-kube-api-access-scqm6\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-oauth-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.681209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.681207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782152 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-oauth-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782152 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-trusted-ca-bundle\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-service-ca\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-console-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782220    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqm6\" (UniqueName: \"kubernetes.io/projected/134d2744-9502-4faa-8c43-db5565cda735-kube-api-access-scqm6\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-oauth-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.782986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-service-ca\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.783256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.782959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-oauth-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.783256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.783052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-console-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.783256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.783126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/134d2744-9502-4faa-8c43-db5565cda735-trusted-ca-bundle\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.784916 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.784896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-serving-cert\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.785040 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.784937    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/134d2744-9502-4faa-8c43-db5565cda735-console-oauth-config\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.791211 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.791186    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqm6\" (UniqueName: \"kubernetes.io/projected/134d2744-9502-4faa-8c43-db5565cda735-kube-api-access-scqm6\") pod \"console-55c5f847b8-pfdfg\" (UID: \"134d2744-9502-4faa-8c43-db5565cda735\") " pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:38.910928 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:38.910827    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:39.184452 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:39.184194    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c5f847b8-pfdfg"]
Sep 09 12:17:39.187589 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:17:39.187544    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134d2744_9502_4faa_8c43_db5565cda735.slice/crio-ceaeb478c0caf4e5466645a33c0abcdc10dbff9d82bed087e7701cba730415c1 WatchSource:0}: Error finding container ceaeb478c0caf4e5466645a33c0abcdc10dbff9d82bed087e7701cba730415c1: Status 404 returned error can't find the container with id ceaeb478c0caf4e5466645a33c0abcdc10dbff9d82bed087e7701cba730415c1
Sep 09 12:17:39.409199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:39.409166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c5f847b8-pfdfg" event={"ID":"134d2744-9502-4faa-8c43-db5565cda735","Type":"ContainerStarted","Data":"413abde893370becac8b6ff7b5dd95c6344d5f120236e3f7f03398dcb4cdf502"}
Sep 09 12:17:39.409381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:39.409213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c5f847b8-pfdfg" event={"ID":"134d2744-9502-4faa-8c43-db5565cda735","Type":"ContainerStarted","Data":"ceaeb478c0caf4e5466645a33c0abcdc10dbff9d82bed087e7701cba730415c1"}
Sep 09 12:17:39.429752 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:39.429685    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55c5f847b8-pfdfg" podStartSLOduration=1.42966716 podStartE2EDuration="1.42966716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:17:39.429145662 +0000 UTC m=+382.059733438" watchObservedRunningTime="2026-09-09 12:17:39.42966716 +0000 UTC m=+382.060254914"
Sep 09 12:17:40.021912 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.021876    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"]
Sep 09 12:17:40.024972 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.024954    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.027697 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.027600    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 12:17:40.027697 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.027625    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 12:17:40.027697 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.027636    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 12:17:40.027697 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.027677    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 12:17:40.027956 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.027746    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ssn29\""
Sep 09 12:17:40.038721 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.038695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"]
Sep 09 12:17:40.091286 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.091201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2ch\" (UniqueName: \"kubernetes.io/projected/dc097444-537d-4563-923d-34b4bfb29a60-kube-api-access-hj2ch\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.091489 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.091327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-tmp\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.091489 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.091395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-manifests\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.091489 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.091451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.091629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.091515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.192558 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.192520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.192760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.192600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2ch\" (UniqueName: \"kubernetes.io/projected/dc097444-537d-4563-923d-34b4bfb29a60-kube-api-access-hj2ch\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.192760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.192626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-tmp\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.192760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.192655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-manifests\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.192760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.192687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.193156 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.193130    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-manifests\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.195291 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.195260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc097444-537d-4563-923d-34b4bfb29a60-tmp\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.195424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.195321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.195424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.195391    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc097444-537d-4563-923d-34b4bfb29a60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.205547 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.205516    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2ch\" (UniqueName: \"kubernetes.io/projected/dc097444-537d-4563-923d-34b4bfb29a60-kube-api-access-hj2ch\") pod \"opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj\" (UID: \"dc097444-537d-4563-923d-34b4bfb29a60\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.335841 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.335747    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:40.615096 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:40.615004    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"]
Sep 09 12:17:40.618098 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:17:40.618069    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc097444_537d_4563_923d_34b4bfb29a60.slice/crio-619669e9741163786282e719b310abd3e4047bfd4ce55c298ed75cf4480096e3 WatchSource:0}: Error finding container 619669e9741163786282e719b310abd3e4047bfd4ce55c298ed75cf4480096e3: Status 404 returned error can't find the container with id 619669e9741163786282e719b310abd3e4047bfd4ce55c298ed75cf4480096e3
Sep 09 12:17:41.419401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:41.419370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj" event={"ID":"dc097444-537d-4563-923d-34b4bfb29a60","Type":"ContainerStarted","Data":"619669e9741163786282e719b310abd3e4047bfd4ce55c298ed75cf4480096e3"}
Sep 09 12:17:44.433864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:44.433821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj" event={"ID":"dc097444-537d-4563-923d-34b4bfb29a60","Type":"ContainerStarted","Data":"fb78718e6097d144eb4e457e34cbc2daf6e7a5a462015fba22e8dbd8e2b79859"}
Sep 09 12:17:45.439300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:45.439259    2556 generic.go:356] "Generic (PLEG): container finished" podID="dc097444-537d-4563-923d-34b4bfb29a60" containerID="fb78718e6097d144eb4e457e34cbc2daf6e7a5a462015fba22e8dbd8e2b79859" exitCode=0
Sep 09 12:17:45.439698 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:45.439339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj" event={"ID":"dc097444-537d-4563-923d-34b4bfb29a60","Type":"ContainerDied","Data":"fb78718e6097d144eb4e457e34cbc2daf6e7a5a462015fba22e8dbd8e2b79859"}
Sep 09 12:17:46.445252 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:46.445204    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj" event={"ID":"dc097444-537d-4563-923d-34b4bfb29a60","Type":"ContainerStarted","Data":"de7f8cf36ad0ecf919c517a4efa8733a4bd3b035b0b26dfcfd3e52b761cd1255"}
Sep 09 12:17:46.445654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:46.445334    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:17:46.470059 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:46.470010    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj" podStartSLOduration=1.9886516429999999 podStartE2EDuration="6.469994942s" totalImagesPullingTime="3.473634371s" totalInitContainerRuntime="1.007708928s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:17:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:17:46.468013674 +0000 UTC m=+389.098601450" watchObservedRunningTime="2026-09-09 12:17:46.469994942 +0000 UTC m=+389.100582717"
Sep 09 12:17:48.911587 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:48.911542    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:48.911986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:48.911618    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:48.916147 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:48.916123    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:49.459335 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:49.459301    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55c5f847b8-pfdfg"
Sep 09 12:17:57.451803 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:17:57.451768    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj"
Sep 09 12:18:05.098824 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.098792    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"]
Sep 09 12:18:05.108896 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.108868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.114616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.114585    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 12:18:05.114616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.114585    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 12:18:05.120734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.120707    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"]
Sep 09 12:18:05.196913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.196875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tmp\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.196913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.196918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p5m\" (UniqueName: \"kubernetes.io/projected/a6f971be-c0d0-4649-91ec-e87a40aea4f5-kube-api-access-96p5m\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.197123 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.196959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tls-certs\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.297410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.297377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tls-certs\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.297575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.297451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tmp\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.297575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.297484    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p5m\" (UniqueName: \"kubernetes.io/projected/a6f971be-c0d0-4649-91ec-e87a40aea4f5-kube-api-access-96p5m\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.299953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.299925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tmp\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.300265 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.300222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f971be-c0d0-4649-91ec-e87a40aea4f5-tls-certs\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.308117 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.308089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p5m\" (UniqueName: \"kubernetes.io/projected/a6f971be-c0d0-4649-91ec-e87a40aea4f5-kube-api-access-96p5m\") pod \"kube-auth-proxy-9b9df8cfb-ncf29\" (UID: \"a6f971be-c0d0-4649-91ec-e87a40aea4f5\") " pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.422498 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.422409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"
Sep 09 12:18:05.696907 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:05.696736    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29"]
Sep 09 12:18:05.699436 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:18:05.699407    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f971be_c0d0_4649_91ec_e87a40aea4f5.slice/crio-cd6b0fb1e9b1f1247128c16913e795c8c7fafa5a09dfd1d5b11548110e626d96 WatchSource:0}: Error finding container cd6b0fb1e9b1f1247128c16913e795c8c7fafa5a09dfd1d5b11548110e626d96: Status 404 returned error can't find the container with id cd6b0fb1e9b1f1247128c16913e795c8c7fafa5a09dfd1d5b11548110e626d96
Sep 09 12:18:06.406719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.406684    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:06.411378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.411359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.414773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.414744    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 12:18:06.414773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.414767    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 12:18:06.414971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.414767    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 12:18:06.415073 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.415057    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 12:18:06.415147 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.415090    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lfhgf\""
Sep 09 12:18:06.425055 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.425027    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:06.505758 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505722    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.505935 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.505935 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5fn\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.505935 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.506056 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.506056 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.506056 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.505992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.522379 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.522348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29" event={"ID":"a6f971be-c0d0-4649-91ec-e87a40aea4f5","Type":"ContainerStarted","Data":"cd6b0fb1e9b1f1247128c16913e795c8c7fafa5a09dfd1d5b11548110e626d96"}
Sep 09 12:18:06.607120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5fn\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607196    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607220    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607402 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607303    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.607983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.607950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.609753 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.609735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.609853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.609795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.610130 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.610115    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.610190 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.610158    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.617074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.617050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5fn\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.617191 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.617164    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p5zcf\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:06.721343 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:06.721300    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:07.017694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:07.017564    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:07.021136 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:18:07.021106    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a95ce6_4ba4_4d03_ac3f_1f22307d8c0d.slice/crio-9ed994c22e097be409dc385963baff25b2c682389e49f93ba2c366809778085e WatchSource:0}: Error finding container 9ed994c22e097be409dc385963baff25b2c682389e49f93ba2c366809778085e: Status 404 returned error can't find the container with id 9ed994c22e097be409dc385963baff25b2c682389e49f93ba2c366809778085e
Sep 09 12:18:07.529176 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:07.529118    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" event={"ID":"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d","Type":"ContainerStarted","Data":"9ed994c22e097be409dc385963baff25b2c682389e49f93ba2c366809778085e"}
Sep 09 12:18:09.539938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:09.539892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29" event={"ID":"a6f971be-c0d0-4649-91ec-e87a40aea4f5","Type":"ContainerStarted","Data":"82caa5b4934f75b2539c7cfe8f08ecd9c88a69cea4144100122785a435672521"}
Sep 09 12:18:09.557642 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:09.557582    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9b9df8cfb-ncf29" podStartSLOduration=1.449158766 podStartE2EDuration="4.557566431s" totalImagesPullingTime="3.108407665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:18:09.556436977 +0000 UTC m=+412.187024754" watchObservedRunningTime="2026-09-09 12:18:09.557566431 +0000 UTC m=+412.188154207"
Sep 09 12:18:11.319827 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:11.319774    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 12:18:11.320263 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:11.319862    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 12:18:11.550445 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:11.550413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" event={"ID":"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d","Type":"ContainerStarted","Data":"b2cfc738782f0976c406aa9048f72fbf944b0bdb517ef8790e4a5b925efa6bf9"}
Sep 09 12:18:11.550639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:11.550534    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:11.572420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:11.572327    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" podStartSLOduration=1.276502456 podStartE2EDuration="5.572310472s" totalImagesPullingTime="4.295808016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:18:11.570347225 +0000 UTC m=+414.200935001" watchObservedRunningTime="2026-09-09 12:18:11.572310472 +0000 UTC m=+414.202898248"
Sep 09 12:18:12.556470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:12.556443    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:19.254067 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.254027    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"]
Sep 09 12:18:19.257165 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.257149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.264300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.264278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8qz6n\""
Sep 09 12:18:19.264800 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.264782    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 12:18:19.265032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.265009    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 12:18:19.265133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.265082    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 12:18:19.265195 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.265144    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 12:18:19.265398 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.265382    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 12:18:19.285171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.285147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"]
Sep 09 12:18:19.424000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.423964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-metrics-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.424000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.424002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc85x\" (UniqueName: \"kubernetes.io/projected/c01f7f7f-9746-4f81-9437-db2ed9639f21-kube-api-access-dc85x\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.424207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.424022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c01f7f7f-9746-4f81-9437-db2ed9639f21-manager-config\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.424207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.424072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.525438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.525325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-metrics-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.525438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.525379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc85x\" (UniqueName: \"kubernetes.io/projected/c01f7f7f-9746-4f81-9437-db2ed9639f21-kube-api-access-dc85x\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.525438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.525398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c01f7f7f-9746-4f81-9437-db2ed9639f21-manager-config\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.525438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.525414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.525996 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.525973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c01f7f7f-9746-4f81-9437-db2ed9639f21-manager-config\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.528005 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.527983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.528095 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.528007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f7f7f-9746-4f81-9437-db2ed9639f21-metrics-cert\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.545411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.545378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc85x\" (UniqueName: \"kubernetes.io/projected/c01f7f7f-9746-4f81-9437-db2ed9639f21-kube-api-access-dc85x\") pod \"lws-controller-manager-df65498f9-v22bh\" (UID: \"c01f7f7f-9746-4f81-9437-db2ed9639f21\") " pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.566818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.566786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:19.851113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:19.847356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"]
Sep 09 12:18:20.588231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:20.588197    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh" event={"ID":"c01f7f7f-9746-4f81-9437-db2ed9639f21","Type":"ContainerStarted","Data":"dc4284bf069f61eda1f3e369acf549d07b71664d121994ebd272de1e3f8b9611"}
Sep 09 12:18:22.598087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:22.598052    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh" event={"ID":"c01f7f7f-9746-4f81-9437-db2ed9639f21","Type":"ContainerStarted","Data":"43765932987d1ff04061da86057577380728e210e1696d823b811ac01815bf6d"}
Sep 09 12:18:22.598528 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:22.598283    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:18:22.639048 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:22.638990    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh" podStartSLOduration=1.080441876 podStartE2EDuration="3.638973997s" totalImagesPullingTime="2.558532121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:18:22.635480012 +0000 UTC m=+425.266067788" watchObservedRunningTime="2026-09-09 12:18:22.638973997 +0000 UTC m=+425.269561773"
Sep 09 12:18:25.437063 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.437024    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:25.437540 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.437285    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" podUID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" containerName="discovery" containerID="cri-o://b2cfc738782f0976c406aa9048f72fbf944b0bdb517ef8790e4a5b925efa6bf9" gracePeriod=30
Sep 09 12:18:25.617197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.617167    2556 generic.go:356] "Generic (PLEG): container finished" podID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" containerID="b2cfc738782f0976c406aa9048f72fbf944b0bdb517ef8790e4a5b925efa6bf9" exitCode=0
Sep 09 12:18:25.617375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.617218    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" event={"ID":"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d","Type":"ContainerDied","Data":"b2cfc738782f0976c406aa9048f72fbf944b0bdb517ef8790e4a5b925efa6bf9"}
Sep 09 12:18:25.669602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.669578    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:25.680100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680070    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680106    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680141    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p5fn\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680156    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680183    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680212    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.680657 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.680621    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:18:25.682515 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.682491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:18:25.682663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.682546    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn" (OuterVolumeSpecName: "kube-api-access-9p5fn") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "kube-api-access-9p5fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:18:25.682663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.682619    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:18:25.682924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.682899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:18:25.682924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.682906    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs" (OuterVolumeSpecName: "local-certs") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:18:25.780806 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780763    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts\") pod \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\" (UID: \"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d\") "
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780906    2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-kubeconfig\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780919    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-dns-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780928    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p5fn\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-kube-api-access-9p5fn\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780938    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-token\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780947    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-istio-csr-ca-configmap\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.781003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.780955    2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-local-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:25.783058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.783032    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts" (OuterVolumeSpecName: "cacerts") pod "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" (UID: "37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:18:25.881899 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:25.881865    2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d-cacerts\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:18:26.623211 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:26.623180    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"
Sep 09 12:18:26.623653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:26.623185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf" event={"ID":"37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d","Type":"ContainerDied","Data":"9ed994c22e097be409dc385963baff25b2c682389e49f93ba2c366809778085e"}
Sep 09 12:18:26.623653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:26.623314    2556 scope.go:117] "RemoveContainer" containerID="b2cfc738782f0976c406aa9048f72fbf944b0bdb517ef8790e4a5b925efa6bf9"
Sep 09 12:18:26.645190 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:26.645160    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:26.651614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:26.651588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p5zcf"]
Sep 09 12:18:27.900332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:27.900299    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" path="/var/lib/kubelet/pods/37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d/volumes"
Sep 09 12:18:33.604780 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:18:33.604747    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-df65498f9-v22bh"
Sep 09 12:19:41.328177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.328143    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"]
Sep 09 12:19:41.328711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.328438    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" containerName="discovery"
Sep 09 12:19:41.328711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.328451    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" containerName="discovery"
Sep 09 12:19:41.328711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.328511    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a95ce6-4ba4-4d03-ac3f-1f22307d8c0d" containerName="discovery"
Sep 09 12:19:41.330175 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.330156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.332642 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.332613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bjgb5\""
Sep 09 12:19:41.332761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.332614    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 12:19:41.332761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.332685    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 12:19:41.332761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.332676    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 12:19:41.333487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.333473    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 12:19:41.340687 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.340663    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"]
Sep 09 12:19:41.380077 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.380039    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.380077 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.380080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.380334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.380113    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khc6p\" (UniqueName: \"kubernetes.io/projected/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-kube-api-access-khc6p\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.481484 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.481445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khc6p\" (UniqueName: \"kubernetes.io/projected/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-kube-api-access-khc6p\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.481656 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.481538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.481656 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.481564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.481656 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:19:41.481645    2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 12:19:41.481755 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:19:41.481718    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert podName:5d231dc9-5ee7-4417-a9fe-0fd147ab0246 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:41.981692132 +0000 UTC m=+504.612279886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-rg5k6" (UID: "5d231dc9-5ee7-4417-a9fe-0fd147ab0246") : secret "plugin-serving-cert" not found
Sep 09 12:19:41.482206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.482185    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.503112 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.503084    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khc6p\" (UniqueName: \"kubernetes.io/projected/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-kube-api-access-khc6p\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.986029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.985978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:41.988594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:41.988567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d231dc9-5ee7-4417-a9fe-0fd147ab0246-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rg5k6\" (UID: \"5d231dc9-5ee7-4417-a9fe-0fd147ab0246\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:42.240061 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:42.239980    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"
Sep 09 12:19:42.510229 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:42.510160    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6"]
Sep 09 12:19:42.513409 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:19:42.513380    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d231dc9_5ee7_4417_a9fe_0fd147ab0246.slice/crio-fc66446263110924c8d316cf60cdecd36c024129ff513573ab97fd5b76a9a0b6 WatchSource:0}: Error finding container fc66446263110924c8d316cf60cdecd36c024129ff513573ab97fd5b76a9a0b6: Status 404 returned error can't find the container with id fc66446263110924c8d316cf60cdecd36c024129ff513573ab97fd5b76a9a0b6
Sep 09 12:19:42.922321 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:42.922225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6" event={"ID":"5d231dc9-5ee7-4417-a9fe-0fd147ab0246","Type":"ContainerStarted","Data":"fc66446263110924c8d316cf60cdecd36c024129ff513573ab97fd5b76a9a0b6"}
Sep 09 12:19:47.949056 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:47.949011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6" event={"ID":"5d231dc9-5ee7-4417-a9fe-0fd147ab0246","Type":"ContainerStarted","Data":"2e447a72d097c1e324072cacb02d869c27608a1ea4840a7bc796fc2dc105a485"}
Sep 09 12:19:47.987093 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:19:47.986950    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rg5k6" podStartSLOduration=2.10032721 podStartE2EDuration="6.986928837s" totalImagesPullingTime="4.886601627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:19:47.985765166 +0000 UTC m=+510.616352942" watchObservedRunningTime="2026-09-09 12:19:47.986928837 +0000 UTC m=+510.617516614"
Sep 09 12:20:36.273334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.273225    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:20:36.294138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.294093    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:20:36.294317 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.294220    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:20:36.296639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.296614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nr2zn\""
Sep 09 12:20:36.427813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.427777    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qv2v\" (UniqueName: \"kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v\") pod \"authorino-7bcfd8f9cd-l8f2k\" (UID: \"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:20:36.512499 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.512465    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:20:36.515602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.515584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:20:36.520384 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.520359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:20:36.528671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.528593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qv2v\" (UniqueName: \"kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v\") pod \"authorino-7bcfd8f9cd-l8f2k\" (UID: \"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:20:36.537510 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.537477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qv2v\" (UniqueName: \"kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v\") pod \"authorino-7bcfd8f9cd-l8f2k\" (UID: \"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:20:36.604373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.604333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:20:36.629626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.629579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkdg\" (UniqueName: \"kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg\") pod \"authorino-59c974bd48-f7nh2\" (UID: \"d66fde02-621c-465f-8526-c1048975d284\") " pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:20:36.730706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.730664    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkdg\" (UniqueName: \"kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg\") pod \"authorino-59c974bd48-f7nh2\" (UID: \"d66fde02-621c-465f-8526-c1048975d284\") " pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:20:36.740171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.740143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkdg\" (UniqueName: \"kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg\") pod \"authorino-59c974bd48-f7nh2\" (UID: \"d66fde02-621c-465f-8526-c1048975d284\") " pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:20:36.826063 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.825983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:20:36.871982 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:36.871852    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:20:36.874676 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:20:36.874633    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd5ab4ac_ba5d_42c6_9a1c_d2c40b86a694.slice/crio-f2d2ce492a85895ac07409491b6efcb0d155545d2c7ab6bce7532d5ea81201ef WatchSource:0}: Error finding container f2d2ce492a85895ac07409491b6efcb0d155545d2c7ab6bce7532d5ea81201ef: Status 404 returned error can't find the container with id f2d2ce492a85895ac07409491b6efcb0d155545d2c7ab6bce7532d5ea81201ef
Sep 09 12:20:37.097805 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:37.097726    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:20:37.100820 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:20:37.100793    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd66fde02_621c_465f_8526_c1048975d284.slice/crio-af8cde5906d10b1d74c9a798aa08b2159c8e73e9d8228052f59a80aa0154a8f2 WatchSource:0}: Error finding container af8cde5906d10b1d74c9a798aa08b2159c8e73e9d8228052f59a80aa0154a8f2: Status 404 returned error can't find the container with id af8cde5906d10b1d74c9a798aa08b2159c8e73e9d8228052f59a80aa0154a8f2
Sep 09 12:20:37.152612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:37.152569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" event={"ID":"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694","Type":"ContainerStarted","Data":"f2d2ce492a85895ac07409491b6efcb0d155545d2c7ab6bce7532d5ea81201ef"}
Sep 09 12:20:37.153650 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:37.153626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f7nh2" event={"ID":"d66fde02-621c-465f-8526-c1048975d284","Type":"ContainerStarted","Data":"af8cde5906d10b1d74c9a798aa08b2159c8e73e9d8228052f59a80aa0154a8f2"}
Sep 09 12:20:40.169856 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:40.169818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" event={"ID":"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694","Type":"ContainerStarted","Data":"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"}
Sep 09 12:20:40.186516 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:20:40.186441    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" podStartSLOduration=1.657635433 podStartE2EDuration="4.186423465s" totalImagesPullingTime="2.528788032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:40.185556455 +0000 UTC m=+562.816144233" watchObservedRunningTime="2026-09-09 12:20:40.186423465 +0000 UTC m=+562.817011241"
Sep 09 12:21:04.272789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.272750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f7nh2" event={"ID":"d66fde02-621c-465f-8526-c1048975d284","Type":"ContainerStarted","Data":"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"}
Sep 09 12:21:04.289983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.289927    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-f7nh2" podStartSLOduration=1.344925408 podStartE2EDuration="28.289908614s" totalImagesPullingTime="26.944983206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:04.288664922 +0000 UTC m=+586.919252699" watchObservedRunningTime="2026-09-09 12:21:04.289908614 +0000 UTC m=+586.920496391"
Sep 09 12:21:04.318155 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.318077    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:21:04.318340 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.318312    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" podUID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" containerName="authorino" containerID="cri-o://da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357" gracePeriod=30
Sep 09 12:21:04.555861 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.555839    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:21:04.634733 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.634649    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qv2v\" (UniqueName: \"kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v\") pod \"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694\" (UID: \"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694\") "
Sep 09 12:21:04.636953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.636928    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v" (OuterVolumeSpecName: "kube-api-access-7qv2v") pod "bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" (UID: "bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694"). InnerVolumeSpecName "kube-api-access-7qv2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:04.735763 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:04.735732    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qv2v\" (UniqueName: \"kubernetes.io/projected/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694-kube-api-access-7qv2v\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:21:05.278274 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.278214    2556 generic.go:356] "Generic (PLEG): container finished" podID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" containerID="da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357" exitCode=0
Sep 09 12:21:05.278679 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.278284    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k"
Sep 09 12:21:05.278679 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.278300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" event={"ID":"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694","Type":"ContainerDied","Data":"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"}
Sep 09 12:21:05.278679 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.278341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l8f2k" event={"ID":"bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694","Type":"ContainerDied","Data":"f2d2ce492a85895ac07409491b6efcb0d155545d2c7ab6bce7532d5ea81201ef"}
Sep 09 12:21:05.278679 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.278358    2556 scope.go:117] "RemoveContainer" containerID="da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"
Sep 09 12:21:05.287301 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.287285    2556 scope.go:117] "RemoveContainer" containerID="da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"
Sep 09 12:21:05.287535 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:21:05.287516    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357\": container with ID starting with da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357 not found: ID does not exist" containerID="da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"
Sep 09 12:21:05.287587 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.287544    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357"} err="failed to get container status \"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357\": rpc error: code = NotFound desc = could not find container \"da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357\": container with ID starting with da214a3131270961c3ed249611ef7d2cd7be3c3290212e832b856be59a3ec357 not found: ID does not exist"
Sep 09 12:21:05.299936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.299909    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:21:05.304420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.304399    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l8f2k"]
Sep 09 12:21:05.901518 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:05.901483    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" path="/var/lib/kubelet/pods/bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694/volumes"
Sep 09 12:21:06.737273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.737225    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-72c48"]
Sep 09 12:21:06.737672 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.737516    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" containerName="authorino"
Sep 09 12:21:06.737672 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.737530    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" containerName="authorino"
Sep 09 12:21:06.737672 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.737592    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5ab4ac-ba5d-42c6-9a1c-d2c40b86a694" containerName="authorino"
Sep 09 12:21:06.740455 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.740439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:06.750617 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.750596    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-72c48"]
Sep 09 12:21:06.828659 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.828615    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-72c48"]
Sep 09 12:21:06.828865 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:21:06.828847    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-58b9b], unattached volumes=[], failed to process volumes=[kube-api-access-58b9b]: context canceled" pod="kuadrant-system/authorino-557f876f69-72c48" podUID="f24535ef-8d0f-4afa-ae3d-2c92412b6221"
Sep 09 12:21:06.853201 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.853171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58b9b\" (UniqueName: \"kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b\") pod \"authorino-557f876f69-72c48\" (UID: \"f24535ef-8d0f-4afa-ae3d-2c92412b6221\") " pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:06.855910 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.855890    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ls6dz"]
Sep 09 12:21:06.858061 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.858046    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:06.860544 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.860525    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 12:21:06.870222 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.870201    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ls6dz"]
Sep 09 12:21:06.953793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.953753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgncg\" (UniqueName: \"kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:06.953988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.953809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58b9b\" (UniqueName: \"kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b\") pod \"authorino-557f876f69-72c48\" (UID: \"f24535ef-8d0f-4afa-ae3d-2c92412b6221\") " pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:06.953988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.953845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:06.963559 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:06.963530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58b9b\" (UniqueName: \"kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b\") pod \"authorino-557f876f69-72c48\" (UID: \"f24535ef-8d0f-4afa-ae3d-2c92412b6221\") " pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:07.003311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.003211    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ls6dz"]
Sep 09 12:21:07.003508 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:21:07.003484    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kgncg tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-ls6dz" podUID="f48ac4fc-3709-4d57-b8fb-e9d794a790b1"
Sep 09 12:21:07.032352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.032322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:21:07.034588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.034574    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.043810 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.043785    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:21:07.054371 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.054350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.054471 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.054379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrrw\" (UniqueName: \"kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.054471 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.054414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgncg\" (UniqueName: \"kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.054557 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.054498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.057060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.057039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.063259 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.063217    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgncg\" (UniqueName: \"kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg\") pod \"authorino-6d4dfc649f-ls6dz\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") " pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.155184 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.155149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.155184 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.155182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrrw\" (UniqueName: \"kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.157677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.157653    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.165215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.165195    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrrw\" (UniqueName: \"kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw\") pod \"authorino-84bc6b6cdb-99kcm\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") " pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.287980 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.287905    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:07.288106 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.287905    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.294386 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.294369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:07.296727 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.296712    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:07.343369 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.343340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:21:07.356420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.356398    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58b9b\" (UniqueName: \"kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b\") pod \"f24535ef-8d0f-4afa-ae3d-2c92412b6221\" (UID: \"f24535ef-8d0f-4afa-ae3d-2c92412b6221\") "
Sep 09 12:21:07.356528 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.356433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert\") pod \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") "
Sep 09 12:21:07.356528 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.356475    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgncg\" (UniqueName: \"kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg\") pod \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\" (UID: \"f48ac4fc-3709-4d57-b8fb-e9d794a790b1\") "
Sep 09 12:21:07.358682 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.358651    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f48ac4fc-3709-4d57-b8fb-e9d794a790b1" (UID: "f48ac4fc-3709-4d57-b8fb-e9d794a790b1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:21:07.358770 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.358682    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b" (OuterVolumeSpecName: "kube-api-access-58b9b") pod "f24535ef-8d0f-4afa-ae3d-2c92412b6221" (UID: "f24535ef-8d0f-4afa-ae3d-2c92412b6221"). InnerVolumeSpecName "kube-api-access-58b9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:07.358934 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.358911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg" (OuterVolumeSpecName: "kube-api-access-kgncg") pod "f48ac4fc-3709-4d57-b8fb-e9d794a790b1" (UID: "f48ac4fc-3709-4d57-b8fb-e9d794a790b1"). InnerVolumeSpecName "kube-api-access-kgncg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:07.457816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.457775    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58b9b\" (UniqueName: \"kubernetes.io/projected/f24535ef-8d0f-4afa-ae3d-2c92412b6221-kube-api-access-58b9b\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:21:07.457816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.457810    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-tls-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:21:07.457816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.457828    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgncg\" (UniqueName: \"kubernetes.io/projected/f48ac4fc-3709-4d57-b8fb-e9d794a790b1-kube-api-access-kgncg\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:21:07.613911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:07.613806    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:21:07.617500 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:21:07.617475    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c1ae7a_76ac_4482_91b3_5ba089fcd950.slice/crio-a0c9e5a4dc50c2e6025d59faee3725369ead0c12173c1f4cba4e4b85602ca572 WatchSource:0}: Error finding container a0c9e5a4dc50c2e6025d59faee3725369ead0c12173c1f4cba4e4b85602ca572: Status 404 returned error can't find the container with id a0c9e5a4dc50c2e6025d59faee3725369ead0c12173c1f4cba4e4b85602ca572
Sep 09 12:21:08.293837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.293801    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" event={"ID":"a1c1ae7a-76ac-4482-91b3-5ba089fcd950","Type":"ContainerStarted","Data":"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"}
Sep 09 12:21:08.293837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.293844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" event={"ID":"a1c1ae7a-76ac-4482-91b3-5ba089fcd950","Type":"ContainerStarted","Data":"a0c9e5a4dc50c2e6025d59faee3725369ead0c12173c1f4cba4e4b85602ca572"}
Sep 09 12:21:08.294366 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.293924    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-72c48"
Sep 09 12:21:08.294366 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.294009    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ls6dz"
Sep 09 12:21:08.310004 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.309959    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" podStartSLOduration=0.847358761 podStartE2EDuration="1.309945875s" totalImagesPullingTime="462.587114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:08.308566468 +0000 UTC m=+590.939154242" watchObservedRunningTime="2026-09-09 12:21:08.309945875 +0000 UTC m=+590.940533683"
Sep 09 12:21:08.336905 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.336871    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ls6dz"]
Sep 09 12:21:08.339458 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.339431    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:21:08.339646 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.339623    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-f7nh2" podUID="d66fde02-621c-465f-8526-c1048975d284" containerName="authorino" containerID="cri-o://79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7" gracePeriod=30
Sep 09 12:21:08.342949 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.342923    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ls6dz"]
Sep 09 12:21:08.365110 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.365078    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-72c48"]
Sep 09 12:21:08.369519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.369493    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-72c48"]
Sep 09 12:21:08.567812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.567788    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:21:08.666187 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.666153    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktkdg\" (UniqueName: \"kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg\") pod \"d66fde02-621c-465f-8526-c1048975d284\" (UID: \"d66fde02-621c-465f-8526-c1048975d284\") "
Sep 09 12:21:08.668364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.668342    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg" (OuterVolumeSpecName: "kube-api-access-ktkdg") pod "d66fde02-621c-465f-8526-c1048975d284" (UID: "d66fde02-621c-465f-8526-c1048975d284"). InnerVolumeSpecName "kube-api-access-ktkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:21:08.767208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:08.767174    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkdg\" (UniqueName: \"kubernetes.io/projected/d66fde02-621c-465f-8526-c1048975d284-kube-api-access-ktkdg\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:21:09.299517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.299438    2556 generic.go:356] "Generic (PLEG): container finished" podID="d66fde02-621c-465f-8526-c1048975d284" containerID="79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7" exitCode=0
Sep 09 12:21:09.299517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.299477    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f7nh2"
Sep 09 12:21:09.299517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.299514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f7nh2" event={"ID":"d66fde02-621c-465f-8526-c1048975d284","Type":"ContainerDied","Data":"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"}
Sep 09 12:21:09.299984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.299544    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f7nh2" event={"ID":"d66fde02-621c-465f-8526-c1048975d284","Type":"ContainerDied","Data":"af8cde5906d10b1d74c9a798aa08b2159c8e73e9d8228052f59a80aa0154a8f2"}
Sep 09 12:21:09.299984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.299560    2556 scope.go:117] "RemoveContainer" containerID="79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"
Sep 09 12:21:09.307847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.307831    2556 scope.go:117] "RemoveContainer" containerID="79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"
Sep 09 12:21:09.308103 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:21:09.308087    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7\": container with ID starting with 79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7 not found: ID does not exist" containerID="79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"
Sep 09 12:21:09.308146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.308112    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7"} err="failed to get container status \"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7\": rpc error: code = NotFound desc = could not find container \"79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7\": container with ID starting with 79dd3ff13b0e774e9441242ffa093d03ec96ee13ca3a3bd3234c38512628d8b7 not found: ID does not exist"
Sep 09 12:21:09.320055 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.320028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:21:09.323602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.323573    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f7nh2"]
Sep 09 12:21:09.901033 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.901003    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66fde02-621c-465f-8526-c1048975d284" path="/var/lib/kubelet/pods/d66fde02-621c-465f-8526-c1048975d284/volumes"
Sep 09 12:21:09.901333 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.901319    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24535ef-8d0f-4afa-ae3d-2c92412b6221" path="/var/lib/kubelet/pods/f24535ef-8d0f-4afa-ae3d-2c92412b6221/volumes"
Sep 09 12:21:09.901531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:09.901520    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48ac4fc-3709-4d57-b8fb-e9d794a790b1" path="/var/lib/kubelet/pods/f48ac4fc-3709-4d57-b8fb-e9d794a790b1/volumes"
Sep 09 12:21:17.791972 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:17.791942    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:21:17.792851 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:17.792829    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:21:17.796075 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:17.796057    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:21:17.796821 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:17.796804    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:21:51.253139 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.253104    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"]
Sep 09 12:21:51.253621 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.253391    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66fde02-621c-465f-8526-c1048975d284" containerName="authorino"
Sep 09 12:21:51.253621 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.253402    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66fde02-621c-465f-8526-c1048975d284" containerName="authorino"
Sep 09 12:21:51.253621 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.253458    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66fde02-621c-465f-8526-c1048975d284" containerName="authorino"
Sep 09 12:21:51.256215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.256199    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.258765 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.258746    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 12:21:51.258889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.258750    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-w889h\""
Sep 09 12:21:51.269173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.269147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"]
Sep 09 12:21:51.423437 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.423397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqkk\" (UniqueName: \"kubernetes.io/projected/19cafb7c-bac5-4114-b0f4-d709037af1bc-kube-api-access-sfqkk\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.423631 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.423454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cafb7c-bac5-4114-b0f4-d709037af1bc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.524971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.524876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqkk\" (UniqueName: \"kubernetes.io/projected/19cafb7c-bac5-4114-b0f4-d709037af1bc-kube-api-access-sfqkk\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.524971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.524927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cafb7c-bac5-4114-b0f4-d709037af1bc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.525733 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.525704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cafb7c-bac5-4114-b0f4-d709037af1bc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.536067 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.536034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqkk\" (UniqueName: \"kubernetes.io/projected/19cafb7c-bac5-4114-b0f4-d709037af1bc-kube-api-access-sfqkk\") pod \"payload-pre-processing-59fbc47bf8-xj6c6\" (UID: \"19cafb7c-bac5-4114-b0f4-d709037af1bc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.566130 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.566089    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:21:51.842060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:51.841957    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"]
Sep 09 12:21:51.844353 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:21:51.844319    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cafb7c_bac5_4114_b0f4_d709037af1bc.slice/crio-e1a8f106cc2f3d81032615337c229491badbe329f79dba766342dcd28cd63646 WatchSource:0}: Error finding container e1a8f106cc2f3d81032615337c229491badbe329f79dba766342dcd28cd63646: Status 404 returned error can't find the container with id e1a8f106cc2f3d81032615337c229491badbe329f79dba766342dcd28cd63646
Sep 09 12:21:52.481253 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:52.481184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6" event={"ID":"19cafb7c-bac5-4114-b0f4-d709037af1bc","Type":"ContainerStarted","Data":"e1a8f106cc2f3d81032615337c229491badbe329f79dba766342dcd28cd63646"}
Sep 09 12:21:54.491224 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:54.491190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6" event={"ID":"19cafb7c-bac5-4114-b0f4-d709037af1bc","Type":"ContainerStarted","Data":"1f3438551d97ed45678e831deb3abf59d81339fa57f3c45c3cc6ee712b7ef564"}
Sep 09 12:21:54.509716 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:54.509659    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6" podStartSLOduration=1.805045696 podStartE2EDuration="3.509644921s" totalImagesPullingTime="1.704599225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:21:54.50847467 +0000 UTC m=+637.139062445" watchObservedRunningTime="2026-09-09 12:21:54.509644921 +0000 UTC m=+637.140232697"
Sep 09 12:21:55.496149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:21:55.496117    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:22:06.502885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:06.502850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xj6c6"
Sep 09 12:22:23.793105 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.793071    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"]
Sep 09 12:22:23.796455 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.796429    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.799251 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.799215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 12:22:23.799385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.799215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 12:22:23.800123 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.800105    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6shtk\""
Sep 09 12:22:23.800288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.800120    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 12:22:23.804254 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.804217    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"]
Sep 09 12:22:23.897137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897105    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.897312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897161    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5f1fc-0747-4b10-953e-5a90445533bf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.897312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqhr\" (UniqueName: \"kubernetes.io/projected/ccb5f1fc-0747-4b10-953e-5a90445533bf-kube-api-access-bwqhr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.897312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897216    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.897312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.897312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.897287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998065    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5f1fc-0747-4b10-953e-5a90445533bf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqhr\" (UniqueName: \"kubernetes.io/projected/ccb5f1fc-0747-4b10-953e-5a90445533bf-kube-api-access-bwqhr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998529    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:23.998579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:23.998546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:24.000711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.000684    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5f1fc-0747-4b10-953e-5a90445533bf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:24.000861 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.000845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5f1fc-0747-4b10-953e-5a90445533bf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:24.011169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.011148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqhr\" (UniqueName: \"kubernetes.io/projected/ccb5f1fc-0747-4b10-953e-5a90445533bf-kube-api-access-bwqhr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4775t\" (UID: \"ccb5f1fc-0747-4b10-953e-5a90445533bf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:24.107427 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.107337    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:24.378690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.378666    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"]
Sep 09 12:22:24.380812 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:22:24.380785    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb5f1fc_0747_4b10_953e_5a90445533bf.slice/crio-908be1a588828375317465092ab7aab3be3bd2026da8856c117c0d4d34fd6c32 WatchSource:0}: Error finding container 908be1a588828375317465092ab7aab3be3bd2026da8856c117c0d4d34fd6c32: Status 404 returned error can't find the container with id 908be1a588828375317465092ab7aab3be3bd2026da8856c117c0d4d34fd6c32
Sep 09 12:22:24.382038 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.382021    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:22:24.622783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:24.622745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t" event={"ID":"ccb5f1fc-0747-4b10-953e-5a90445533bf","Type":"ContainerStarted","Data":"908be1a588828375317465092ab7aab3be3bd2026da8856c117c0d4d34fd6c32"}
Sep 09 12:22:25.628904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:25.628869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t" event={"ID":"ccb5f1fc-0747-4b10-953e-5a90445533bf","Type":"ContainerStarted","Data":"e55f50bccb0c2933f1844877a63c628b43af61fa3879a4d6dca734e4b2b39043"}
Sep 09 12:22:25.648575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:25.648518    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t" podStartSLOduration=1.5815016370000001 podStartE2EDuration="2.648502194s" totalImagesPullingTime="1.067000557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:22:25.646949886 +0000 UTC m=+668.277537675" watchObservedRunningTime="2026-09-09 12:22:25.648502194 +0000 UTC m=+668.279089969"
Sep 09 12:22:34.108013 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:34.107961    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:34.108013 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:34.108029    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:34.120483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:34.120451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:34.675911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:34.675884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4775t"
Sep 09 12:22:37.212924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.212895    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"]
Sep 09 12:22:37.216705 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.216688    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.219081 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.219054    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 12:22:37.223940 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.223918    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"]
Sep 09 12:22:37.310355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.310307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.310554 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.310364    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.310719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.310699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3657b2-a1c6-4a7e-830e-081774bf803a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.310963 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.310943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrbs\" (UniqueName: \"kubernetes.io/projected/4f3657b2-a1c6-4a7e-830e-081774bf803a-kube-api-access-4mrbs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.311143 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.311128    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.311284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.311269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3657b2-a1c6-4a7e-830e-081774bf803a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412341    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrbs\" (UniqueName: \"kubernetes.io/projected/4f3657b2-a1c6-4a7e-830e-081774bf803a-kube-api-access-4mrbs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412416    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412576 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412576 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412733 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.412793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.412736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.414713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.414692    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3657b2-a1c6-4a7e-830e-081774bf803a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.414965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.414948    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3657b2-a1c6-4a7e-830e-081774bf803a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.421092 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.421067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrbs\" (UniqueName: \"kubernetes.io/projected/4f3657b2-a1c6-4a7e-830e-081774bf803a-kube-api-access-4mrbs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr\" (UID: \"4f3657b2-a1c6-4a7e-830e-081774bf803a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.526859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.526758    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:37.797130 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:37.797051    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"]
Sep 09 12:22:37.799919 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:22:37.799892    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3657b2_a1c6_4a7e_830e_081774bf803a.slice/crio-6b83f84d989713907d91192ea567961f92d69c8b9ceacb47faa773ce203ea44e WatchSource:0}: Error finding container 6b83f84d989713907d91192ea567961f92d69c8b9ceacb47faa773ce203ea44e: Status 404 returned error can't find the container with id 6b83f84d989713907d91192ea567961f92d69c8b9ceacb47faa773ce203ea44e
Sep 09 12:22:38.682598 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:38.682562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr" event={"ID":"4f3657b2-a1c6-4a7e-830e-081774bf803a","Type":"ContainerStarted","Data":"555db07ca89a01f20aec55a537ebbd9354a763f6a557e5118c38ea70e5e38b8c"}
Sep 09 12:22:38.683072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:38.682625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr" event={"ID":"4f3657b2-a1c6-4a7e-830e-081774bf803a","Type":"ContainerStarted","Data":"6b83f84d989713907d91192ea567961f92d69c8b9ceacb47faa773ce203ea44e"}
Sep 09 12:22:38.699835 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:38.699789    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr" podStartSLOduration=1.499231683 podStartE2EDuration="1.699774271s" totalImagesPullingTime="200.542588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:22:38.699356201 +0000 UTC m=+681.329943977" watchObservedRunningTime="2026-09-09 12:22:38.699774271 +0000 UTC m=+681.330362046"
Sep 09 12:22:47.527439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:47.527398    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:47.527842 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:47.527464    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:47.539923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:47.539899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:22:47.731681 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:22:47.731647    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr"
Sep 09 12:23:00.879829 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.879798    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"]
Sep 09 12:23:00.884066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.884050    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.886430 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.886408    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 12:23:00.895917 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.895894    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"]
Sep 09 12:23:00.919006 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.918973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.919133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.919020    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.919133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.919043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569646c-fd4e-4000-8336-ebdd37054075-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.919207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.919157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.919207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.919190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5l68\" (UniqueName: \"kubernetes.io/projected/b569646c-fd4e-4000-8336-ebdd37054075-kube-api-access-w5l68\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:00.919307 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:00.919212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.019646 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019602    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.019827 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569646c-fd4e-4000-8336-ebdd37054075-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.019894 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.019948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5l68\" (UniqueName: \"kubernetes.io/projected/b569646c-fd4e-4000-8336-ebdd37054075-kube-api-access-w5l68\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.019948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.020058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019973    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.020058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.019998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.020316 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.020292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.020405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.020336    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.022215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.022192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569646c-fd4e-4000-8336-ebdd37054075-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.022428 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.022410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569646c-fd4e-4000-8336-ebdd37054075-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.030228 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.030207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5l68\" (UniqueName: \"kubernetes.io/projected/b569646c-fd4e-4000-8336-ebdd37054075-kube-api-access-w5l68\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2\" (UID: \"b569646c-fd4e-4000-8336-ebdd37054075\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.193480 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.193402    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:01.465784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.465744    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"]
Sep 09 12:23:01.467897 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:23:01.467869    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb569646c_fd4e_4000_8336_ebdd37054075.slice/crio-b4c441027f740d2ed1b2498af4269234624ba7e69422392a42e1ee4a274537f8 WatchSource:0}: Error finding container b4c441027f740d2ed1b2498af4269234624ba7e69422392a42e1ee4a274537f8: Status 404 returned error can't find the container with id b4c441027f740d2ed1b2498af4269234624ba7e69422392a42e1ee4a274537f8
Sep 09 12:23:01.779684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:01.779653    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2" event={"ID":"b569646c-fd4e-4000-8336-ebdd37054075","Type":"ContainerStarted","Data":"b4c441027f740d2ed1b2498af4269234624ba7e69422392a42e1ee4a274537f8"}
Sep 09 12:23:02.786928 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:02.786895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2" event={"ID":"b569646c-fd4e-4000-8336-ebdd37054075","Type":"ContainerStarted","Data":"c8900db2ff7c73050ffdd3a580d52d8fdf8585245cc460718f2e99de1021305a"}
Sep 09 12:23:02.803691 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:02.803641    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2" podStartSLOduration=2.550794128 podStartE2EDuration="2.803621336s" totalImagesPullingTime="252.827208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:23:02.802897449 +0000 UTC m=+705.433485224" watchObservedRunningTime="2026-09-09 12:23:02.803621336 +0000 UTC m=+705.434209112"
Sep 09 12:23:11.194275 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:11.194171    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:11.194691 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:11.194272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:11.206676 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:11.206653    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:23:11.838227 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:23:11.838200    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2"
Sep 09 12:24:05.560332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.560297    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8f95fd98c-5zm89"]
Sep 09 12:24:05.563484 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.563468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.571348 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.571324    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8f95fd98c-5zm89"]
Sep 09 12:24:05.671609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.671576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-tls-cert\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.671769 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.671621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrqc\" (UniqueName: \"kubernetes.io/projected/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-kube-api-access-pfrqc\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.773016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.772980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-tls-cert\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.773139 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.773028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrqc\" (UniqueName: \"kubernetes.io/projected/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-kube-api-access-pfrqc\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.775604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.775571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-tls-cert\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.783108 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.783076    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrqc\" (UniqueName: \"kubernetes.io/projected/975d5213-a56b-4b59-8d4b-8a629a1fcdc1-kube-api-access-pfrqc\") pod \"authorino-8f95fd98c-5zm89\" (UID: \"975d5213-a56b-4b59-8d4b-8a629a1fcdc1\") " pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:05.872953 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:05.872877    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8f95fd98c-5zm89"
Sep 09 12:24:06.149423 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:06.149348    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8f95fd98c-5zm89"]
Sep 09 12:24:06.152353 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:06.152326    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod975d5213_a56b_4b59_8d4b_8a629a1fcdc1.slice/crio-7359055018db999c1910d4a9d952be3869c300fc9cf91ffcd34016db4a99bf05 WatchSource:0}: Error finding container 7359055018db999c1910d4a9d952be3869c300fc9cf91ffcd34016db4a99bf05: Status 404 returned error can't find the container with id 7359055018db999c1910d4a9d952be3869c300fc9cf91ffcd34016db4a99bf05
Sep 09 12:24:07.073729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.073453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8f95fd98c-5zm89" event={"ID":"975d5213-a56b-4b59-8d4b-8a629a1fcdc1","Type":"ContainerStarted","Data":"58b190ad1cac187e443bebf8e78b4c9b6a535c4f6cb0bc220b2e0dd6226285ca"}
Sep 09 12:24:07.073729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.073566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8f95fd98c-5zm89" event={"ID":"975d5213-a56b-4b59-8d4b-8a629a1fcdc1","Type":"ContainerStarted","Data":"7359055018db999c1910d4a9d952be3869c300fc9cf91ffcd34016db4a99bf05"}
Sep 09 12:24:07.095748 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.095681    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8f95fd98c-5zm89" podStartSLOduration=1.4495886439999999 podStartE2EDuration="2.09565862s" totalImagesPullingTime="646.069976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:07.094492991 +0000 UTC m=+769.725080770" watchObservedRunningTime="2026-09-09 12:24:07.09565862 +0000 UTC m=+769.726246393"
Sep 09 12:24:07.123096 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.123052    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:24:07.123325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.123298    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" podUID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" containerName="authorino" containerID="cri-o://351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34" gracePeriod=30
Sep 09 12:24:07.389124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.389100    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:24:07.487073 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.487037    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlrrw\" (UniqueName: \"kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw\") pod \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") "
Sep 09 12:24:07.487287 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.487086    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert\") pod \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\" (UID: \"a1c1ae7a-76ac-4482-91b3-5ba089fcd950\") "
Sep 09 12:24:07.489372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.489345    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw" (OuterVolumeSpecName: "kube-api-access-vlrrw") pod "a1c1ae7a-76ac-4482-91b3-5ba089fcd950" (UID: "a1c1ae7a-76ac-4482-91b3-5ba089fcd950"). InnerVolumeSpecName "kube-api-access-vlrrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:24:07.498871 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.498843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a1c1ae7a-76ac-4482-91b3-5ba089fcd950" (UID: "a1c1ae7a-76ac-4482-91b3-5ba089fcd950"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:24:07.588194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.588147    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrrw\" (UniqueName: \"kubernetes.io/projected/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-kube-api-access-vlrrw\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:24:07.588194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:07.588185    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1c1ae7a-76ac-4482-91b3-5ba089fcd950-tls-cert\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:24:08.079760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.079675    2556 generic.go:356] "Generic (PLEG): container finished" podID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" containerID="351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34" exitCode=0
Sep 09 12:24:08.079760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.079727    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm"
Sep 09 12:24:08.080273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.079780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" event={"ID":"a1c1ae7a-76ac-4482-91b3-5ba089fcd950","Type":"ContainerDied","Data":"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"}
Sep 09 12:24:08.080273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.079827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bc6b6cdb-99kcm" event={"ID":"a1c1ae7a-76ac-4482-91b3-5ba089fcd950","Type":"ContainerDied","Data":"a0c9e5a4dc50c2e6025d59faee3725369ead0c12173c1f4cba4e4b85602ca572"}
Sep 09 12:24:08.080273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.079853    2556 scope.go:117] "RemoveContainer" containerID="351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"
Sep 09 12:24:08.088579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.088560    2556 scope.go:117] "RemoveContainer" containerID="351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"
Sep 09 12:24:08.088878 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:24:08.088860    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34\": container with ID starting with 351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34 not found: ID does not exist" containerID="351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"
Sep 09 12:24:08.088942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.088890    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34"} err="failed to get container status \"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34\": rpc error: code = NotFound desc = could not find container \"351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34\": container with ID starting with 351463a2514a9b8c8bdd2f0581ca77212f9f7caca009606e4af17f686eb97d34 not found: ID does not exist"
Sep 09 12:24:08.100025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.099988    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:24:08.106636 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:08.106613    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84bc6b6cdb-99kcm"]
Sep 09 12:24:09.901043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:09.901008    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" path="/var/lib/kubelet/pods/a1c1ae7a-76ac-4482-91b3-5ba089fcd950/volumes"
Sep 09 12:24:30.491269 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.491210    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:30.491868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.491651    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" containerName="authorino"
Sep 09 12:24:30.491868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.491670    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" containerName="authorino"
Sep 09 12:24:30.491868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.491796    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c1ae7a-76ac-4482-91b3-5ba089fcd950" containerName="authorino"
Sep 09 12:24:30.496096 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.496074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:30.498318 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.498300    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kwwl8\""
Sep 09 12:24:30.500862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.500837    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:30.587071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.587026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkf9\" (UniqueName: \"kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9\") pod \"test-curl-7893-140430321155760\" (UID: \"5d759461-7985-4170-a205-aa180de0c95c\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:30.688412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.688347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkf9\" (UniqueName: \"kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9\") pod \"test-curl-7893-140430321155760\" (UID: \"5d759461-7985-4170-a205-aa180de0c95c\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:30.699561 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.699523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkf9\" (UniqueName: \"kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9\") pod \"test-curl-7893-140430321155760\" (UID: \"5d759461-7985-4170-a205-aa180de0c95c\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:30.806822 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:30.806737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:31.079822 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:31.079798    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:31.082284 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:31.082255    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d759461_7985_4170_a205_aa180de0c95c.slice/crio-6dd9a59839c8f661fdaad941cab8c5bf3d18a36b633c4b9915dfb84015782832 WatchSource:0}: Error finding container 6dd9a59839c8f661fdaad941cab8c5bf3d18a36b633c4b9915dfb84015782832: Status 404 returned error can't find the container with id 6dd9a59839c8f661fdaad941cab8c5bf3d18a36b633c4b9915dfb84015782832
Sep 09 12:24:31.191805 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:31.191769    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7893-140430321155760" event={"ID":"5d759461-7985-4170-a205-aa180de0c95c","Type":"ContainerStarted","Data":"6dd9a59839c8f661fdaad941cab8c5bf3d18a36b633c4b9915dfb84015782832"}
Sep 09 12:24:33.203704 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.203671    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7893-140430321155760" event={"ID":"5d759461-7985-4170-a205-aa180de0c95c","Type":"ContainerStarted","Data":"657a4e8e767aea5067ffebd1c1759932ab4c0ad3cae052a72e1937920e5e0e1c"}
Sep 09 12:24:33.221973 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.221928    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7893-140430321155760" podStartSLOduration=1.197471959 podStartE2EDuration="3.221913477s" totalImagesPullingTime="2.024441518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:33.220505656 +0000 UTC m=+795.851093433" watchObservedRunningTime="2026-09-09 12:24:33.221913477 +0000 UTC m=+795.852501252"
Sep 09 12:24:33.340414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.340380    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7893-140430321155760_5d759461-7985-4170-a205-aa180de0c95c/test-curl-7893-140430321155760/0.log"
Sep 09 12:24:33.628775 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.628743    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:24:33.631900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.631885    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.634436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.634411    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e49066\""
Sep 09 12:24:33.640354 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.635079    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e49066-dockercfg-67vb7\""
Sep 09 12:24:33.649655 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.649629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:24:33.672914 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.672882    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:24:33.676153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.676136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.689098 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.689070    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:24:33.717482 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.717444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbzb\" (UniqueName: \"kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.717628 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.717491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.818366 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.818329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbzb\" (UniqueName: \"kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.818555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.818385    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.818555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.818417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnf7\" (UniqueName: \"kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.818555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.818445    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.819066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.819047    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.828145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.828115    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbzb\" (UniqueName: \"kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb\") pod \"payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.919167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.919092    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnf7\" (UniqueName: \"kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.919167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.919129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.919799 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.919780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.941351 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.941322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnf7\" (UniqueName: \"kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7\") pod \"payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:33.947297 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.947277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:33.986413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:33.986378    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:34.209803 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.209772    2556 generic.go:356] "Generic (PLEG): container finished" podID="5d759461-7985-4170-a205-aa180de0c95c" containerID="657a4e8e767aea5067ffebd1c1759932ab4c0ad3cae052a72e1937920e5e0e1c" exitCode=0
Sep 09 12:24:34.210153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.209855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7893-140430321155760" event={"ID":"5d759461-7985-4170-a205-aa180de0c95c","Type":"ContainerDied","Data":"657a4e8e767aea5067ffebd1c1759932ab4c0ad3cae052a72e1937920e5e0e1c"}
Sep 09 12:24:34.226443 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.226421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:24:34.227621 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:34.227594    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c6a4a3_701e_44bf_b781_dfed2f9c27e8.slice/crio-e6595ca2ee9c9c7e97328a09f7f2bf188079b307c9f9ffc173f4b8162778a84a WatchSource:0}: Error finding container e6595ca2ee9c9c7e97328a09f7f2bf188079b307c9f9ffc173f4b8162778a84a: Status 404 returned error can't find the container with id e6595ca2ee9c9c7e97328a09f7f2bf188079b307c9f9ffc173f4b8162778a84a
Sep 09 12:24:34.270405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.270380    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:24:34.273059 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:34.273030    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003e846b_e79f_4e92_9dd2_f73e46c4eb9c.slice/crio-91b8f79741ef11e57daefb8a70d9b6481e9f49cd95c1f768cd04ae3765b326b6 WatchSource:0}: Error finding container 91b8f79741ef11e57daefb8a70d9b6481e9f49cd95c1f768cd04ae3765b326b6: Status 404 returned error can't find the container with id 91b8f79741ef11e57daefb8a70d9b6481e9f49cd95c1f768cd04ae3765b326b6
Sep 09 12:24:34.815923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.815891    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:24:34.819644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.819613    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:34.822324 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.822302    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c554db-dockercfg-thssz\""
Sep 09 12:24:34.822436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.822344    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c554db\""
Sep 09 12:24:34.829117 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.829092    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:24:34.929497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.929461    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d72s\" (UniqueName: \"kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:34.929686 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:34.929591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.030568 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.030520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.030728 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.030611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d72s\" (UniqueName: \"kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.031233 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.031215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.040076 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.040054    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d72s\" (UniqueName: \"kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s\") pod \"payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.131862 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.131785    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:35.216171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.216077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" event={"ID":"003e846b-e79f-4e92-9dd2-f73e46c4eb9c","Type":"ContainerStarted","Data":"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"}
Sep 09 12:24:35.216171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.216138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" event={"ID":"003e846b-e79f-4e92-9dd2-f73e46c4eb9c","Type":"ContainerStarted","Data":"91b8f79741ef11e57daefb8a70d9b6481e9f49cd95c1f768cd04ae3765b326b6"}
Sep 09 12:24:35.216649 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.216262    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:35.218146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.218116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" event={"ID":"66c6a4a3-701e-44bf-b781-dfed2f9c27e8","Type":"ContainerStarted","Data":"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"}
Sep 09 12:24:35.218303 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.218171    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" event={"ID":"66c6a4a3-701e-44bf-b781-dfed2f9c27e8","Type":"ContainerStarted","Data":"e6595ca2ee9c9c7e97328a09f7f2bf188079b307c9f9ffc173f4b8162778a84a"}
Sep 09 12:24:35.240033 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.239894    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" podStartSLOduration=2.239872754 podStartE2EDuration="2.239872754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:35.239582326 +0000 UTC m=+797.870170105" watchObservedRunningTime="2026-09-09 12:24:35.239872754 +0000 UTC m=+797.870460531"
Sep 09 12:24:35.259288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.259205    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" podStartSLOduration=2.25918444 podStartE2EDuration="2.25918444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:35.258958072 +0000 UTC m=+797.889545852" watchObservedRunningTime="2026-09-09 12:24:35.25918444 +0000 UTC m=+797.889772217"
Sep 09 12:24:35.355665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.355638    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.387822 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.387737    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:35.389276 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.389233    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:35.414919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.414892    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:24:35.418088 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:35.418057    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c75f6c6_5e92_47f8_8b38_9a2a3b161770.slice/crio-77aad947a66a0b019b2997011d64b7ca5f2bce849660846d18d324d911c49fe9 WatchSource:0}: Error finding container 77aad947a66a0b019b2997011d64b7ca5f2bce849660846d18d324d911c49fe9: Status 404 returned error can't find the container with id 77aad947a66a0b019b2997011d64b7ca5f2bce849660846d18d324d911c49fe9
Sep 09 12:24:35.519823 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.519789    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:35.520143 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.520129    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d759461-7985-4170-a205-aa180de0c95c" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:35.520187 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.520145    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d759461-7985-4170-a205-aa180de0c95c" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:35.520223 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.520213    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d759461-7985-4170-a205-aa180de0c95c" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:35.523222 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.523206    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.527762 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.527736    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:35.535208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.535188    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghkf9\" (UniqueName: \"kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9\") pod \"5d759461-7985-4170-a205-aa180de0c95c\" (UID: \"5d759461-7985-4170-a205-aa180de0c95c\") "
Sep 09 12:24:35.537593 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.537570    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9" (OuterVolumeSpecName: "kube-api-access-ghkf9") pod "5d759461-7985-4170-a205-aa180de0c95c" (UID: "5d759461-7985-4170-a205-aa180de0c95c"). InnerVolumeSpecName "kube-api-access-ghkf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:24:35.636724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.636692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4sd\" (UniqueName: \"kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd\") pod \"test-curl-7893-140430321155760\" (UID: \"44a0dc9d-498c-4519-b9c5-5255da1829a8\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.636869 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.636820    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghkf9\" (UniqueName: \"kubernetes.io/projected/5d759461-7985-4170-a205-aa180de0c95c-kube-api-access-ghkf9\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:24:35.737968 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.737932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4sd\" (UniqueName: \"kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd\") pod \"test-curl-7893-140430321155760\" (UID: \"44a0dc9d-498c-4519-b9c5-5255da1829a8\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.751957 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.751926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4sd\" (UniqueName: \"kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd\") pod \"test-curl-7893-140430321155760\" (UID: \"44a0dc9d-498c-4519-b9c5-5255da1829a8\") " pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.780392 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.780354    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:24:35.783514 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.783499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:35.786036 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.786015    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bc0270\""
Sep 09 12:24:35.786513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.786492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bc0270-dockercfg-78cjh\""
Sep 09 12:24:35.793888 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.793862    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:24:35.833576 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.833551    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:35.907452 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.907420    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d759461-7985-4170-a205-aa180de0c95c" path="/var/lib/kubelet/pods/5d759461-7985-4170-a205-aa180de0c95c/volumes"
Sep 09 12:24:35.939141 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.939112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:35.939309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:35.939166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stth\" (UniqueName: \"kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.040699 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.040609    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stth\" (UniqueName: \"kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.040875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.040720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.041577 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.041548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.049631 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.049604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stth\" (UniqueName: \"kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth\") pod \"payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.095657 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.095615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:36.110412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.110374    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:36.112906 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:36.112861    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44a0dc9d_498c_4519_b9c5_5255da1829a8.slice/crio-47a785cd1a48f5301d7f4b3e5f14a7e834928598b377f2aeb98e08def474dfed WatchSource:0}: Error finding container 47a785cd1a48f5301d7f4b3e5f14a7e834928598b377f2aeb98e08def474dfed: Status 404 returned error can't find the container with id 47a785cd1a48f5301d7f4b3e5f14a7e834928598b377f2aeb98e08def474dfed
Sep 09 12:24:36.225167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.225137    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:36.225604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.225257    2556 scope.go:117] "RemoveContainer" containerID="657a4e8e767aea5067ffebd1c1759932ab4c0ad3cae052a72e1937920e5e0e1c"
Sep 09 12:24:36.227282 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.227233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7893-140430321155760" event={"ID":"44a0dc9d-498c-4519-b9c5-5255da1829a8","Type":"ContainerStarted","Data":"47a785cd1a48f5301d7f4b3e5f14a7e834928598b377f2aeb98e08def474dfed"}
Sep 09 12:24:36.230057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.229551    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" event={"ID":"3c75f6c6-5e92-47f8-8b38-9a2a3b161770","Type":"ContainerStarted","Data":"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"}
Sep 09 12:24:36.230057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.229638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" event={"ID":"3c75f6c6-5e92-47f8-8b38-9a2a3b161770","Type":"ContainerStarted","Data":"77aad947a66a0b019b2997011d64b7ca5f2bce849660846d18d324d911c49fe9"}
Sep 09 12:24:36.230057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.230016    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:36.230057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.230048    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:36.249637 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.249568    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" podStartSLOduration=2.249551994 podStartE2EDuration="2.249551994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:36.249103897 +0000 UTC m=+798.879691674" watchObservedRunningTime="2026-09-09 12:24:36.249551994 +0000 UTC m=+798.880139772"
Sep 09 12:24:36.374424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:36.374229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:24:36.376759 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:36.376734    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212aeda0_e271_4330_8084_07a9e0b3a0d1.slice/crio-d813e34e1274a24861b8fccf3d1b4bf05117b008cf862f741aed0ff646c239aa WatchSource:0}: Error finding container d813e34e1274a24861b8fccf3d1b4bf05117b008cf862f741aed0ff646c239aa: Status 404 returned error can't find the container with id d813e34e1274a24861b8fccf3d1b4bf05117b008cf862f741aed0ff646c239aa
Sep 09 12:24:37.236457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.236426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" event={"ID":"212aeda0-e271-4330-8084-07a9e0b3a0d1","Type":"ContainerStarted","Data":"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"}
Sep 09 12:24:37.236881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.236482    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" event={"ID":"212aeda0-e271-4330-8084-07a9e0b3a0d1","Type":"ContainerStarted","Data":"d813e34e1274a24861b8fccf3d1b4bf05117b008cf862f741aed0ff646c239aa"}
Sep 09 12:24:37.236881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.236507    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:37.238746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.238718    2556 generic.go:356] "Generic (PLEG): container finished" podID="44a0dc9d-498c-4519-b9c5-5255da1829a8" containerID="b8862c83b023845cd63063ee3467e259d4d51b403806a38dd3fdfaf63ed0b66d" exitCode=0
Sep 09 12:24:37.238851 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.238794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7893-140430321155760" event={"ID":"44a0dc9d-498c-4519-b9c5-5255da1829a8","Type":"ContainerDied","Data":"b8862c83b023845cd63063ee3467e259d4d51b403806a38dd3fdfaf63ed0b66d"}
Sep 09 12:24:37.254984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:37.254940    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" podStartSLOduration=2.254926495 podStartE2EDuration="2.254926495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:37.253584699 +0000 UTC m=+799.884172477" watchObservedRunningTime="2026-09-09 12:24:37.254926495 +0000 UTC m=+799.885514268"
Sep 09 12:24:38.372117 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.372093    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:38.397807 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.397769    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7893-140430321155760_44a0dc9d-498c-4519-b9c5-5255da1829a8/test-curl-7893-140430321155760/0.log"
Sep 09 12:24:38.425435 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.425356    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:38.429473 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.429444    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7893-140430321155760"]
Sep 09 12:24:38.461172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.461148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl4sd\" (UniqueName: \"kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd\") pod \"44a0dc9d-498c-4519-b9c5-5255da1829a8\" (UID: \"44a0dc9d-498c-4519-b9c5-5255da1829a8\") "
Sep 09 12:24:38.463387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.463362    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd" (OuterVolumeSpecName: "kube-api-access-hl4sd") pod "44a0dc9d-498c-4519-b9c5-5255da1829a8" (UID: "44a0dc9d-498c-4519-b9c5-5255da1829a8"). InnerVolumeSpecName "kube-api-access-hl4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:24:38.562767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:38.562681    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4sd\" (UniqueName: \"kubernetes.io/projected/44a0dc9d-498c-4519-b9c5-5255da1829a8-kube-api-access-hl4sd\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:24:39.254983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:39.254946    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a785cd1a48f5301d7f4b3e5f14a7e834928598b377f2aeb98e08def474dfed"
Sep 09 12:24:39.255163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:39.255034    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:39.280252 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:24:39.280192    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7893-140430321155760\" is forbidden: User \"system:node:ip-10-0-128-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-117.ec2.internal' and this object" podUID="44a0dc9d-498c-4519-b9c5-5255da1829a8" pod="opendatahub/test-curl-7893-140430321155760"
Sep 09 12:24:39.901277 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:39.901230    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a0dc9d-498c-4519-b9c5-5255da1829a8" path="/var/lib/kubelet/pods/44a0dc9d-498c-4519-b9c5-5255da1829a8/volumes"
Sep 09 12:24:43.136444 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.136405    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:24:43.136823 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.136744    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a0dc9d-498c-4519-b9c5-5255da1829a8" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:43.136823 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.136756    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a0dc9d-498c-4519-b9c5-5255da1829a8" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:43.136823 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.136820    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a0dc9d-498c-4519-b9c5-5255da1829a8" containerName="test-curl-7893-140430321155760"
Sep 09 12:24:43.139308 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.139293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.141671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.141650    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c801b865\""
Sep 09 12:24:43.141800 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.141688    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c801b865-dockercfg-76xrv\""
Sep 09 12:24:43.147961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.147937    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:24:43.255076 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.255038    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:24:43.258072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.258056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.269597 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.269573    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:24:43.306270 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.306201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.306417 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.306354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjkj\" (UniqueName: \"kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.407723 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.407615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.407723 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.407719    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts6k\" (UniqueName: \"kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.407901 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.407755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.407901 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.407783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjkj\" (UniqueName: \"kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.408444 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.408423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.416843 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.416811    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjkj\" (UniqueName: \"kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj\") pod \"payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.450914 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.450876    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:43.509919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.508875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts6k\" (UniqueName: \"kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.509919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.508931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.509919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.509862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.522791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.519868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts6k\" (UniqueName: \"kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k\") pod \"payload-processing-e2e-ait-c801b865-5554f78b9c-grtds\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") " pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.567694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.567654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:43.723150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.723126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:24:43.725409 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:43.725374    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ae97e6_28a7_42bb_86b8_94584ff8ca64.slice/crio-be3599d4a3689878444af074578a9cca5ea6d4322967482b9c6ad9a91a075caa WatchSource:0}: Error finding container be3599d4a3689878444af074578a9cca5ea6d4322967482b9c6ad9a91a075caa: Status 404 returned error can't find the container with id be3599d4a3689878444af074578a9cca5ea6d4322967482b9c6ad9a91a075caa
Sep 09 12:24:43.838378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:43.838350    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:24:43.840952 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:43.840926    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bfcc67_320b_4aff_8f05_2768033287a3.slice/crio-b7ca6e3130ac3837a8495dcf1972281abf32d47ae94503e6a88c0c6e05090c8c WatchSource:0}: Error finding container b7ca6e3130ac3837a8495dcf1972281abf32d47ae94503e6a88c0c6e05090c8c: Status 404 returned error can't find the container with id b7ca6e3130ac3837a8495dcf1972281abf32d47ae94503e6a88c0c6e05090c8c
Sep 09 12:24:44.280338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.280300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" event={"ID":"43bfcc67-320b-4aff-8f05-2768033287a3","Type":"ContainerStarted","Data":"8ef3488b0dc7e18178bfbb788379d7439ed264109fbc0ac5086efd88311437a0"}
Sep 09 12:24:44.280818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.280367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" event={"ID":"43bfcc67-320b-4aff-8f05-2768033287a3","Type":"ContainerStarted","Data":"b7ca6e3130ac3837a8495dcf1972281abf32d47ae94503e6a88c0c6e05090c8c"}
Sep 09 12:24:44.280818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.280397    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:44.281919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.281889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" event={"ID":"66ae97e6-28a7-42bb-86b8-94584ff8ca64","Type":"ContainerStarted","Data":"5cb827631466e527363515402c7c04ad3eb449c1788eaf4a91ee5a49f49c0a02"}
Sep 09 12:24:44.282023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.281944    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" event={"ID":"66ae97e6-28a7-42bb-86b8-94584ff8ca64","Type":"ContainerStarted","Data":"be3599d4a3689878444af074578a9cca5ea6d4322967482b9c6ad9a91a075caa"}
Sep 09 12:24:44.282023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.281968    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:44.300014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.299958    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" podStartSLOduration=1.299944479 podStartE2EDuration="1.299944479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:44.298574698 +0000 UTC m=+806.929162477" watchObservedRunningTime="2026-09-09 12:24:44.299944479 +0000 UTC m=+806.930532245"
Sep 09 12:24:44.317431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:44.317381    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" podStartSLOduration=1.317365206 podStartE2EDuration="1.317365206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:44.314951177 +0000 UTC m=+806.945538947" watchObservedRunningTime="2026-09-09 12:24:44.317365206 +0000 UTC m=+806.947952982"
Sep 09 12:24:45.204546 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.204504    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:24:45.207876 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.207858    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.210321 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.210298    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a1e1f1-dockercfg-grvv5\""
Sep 09 12:24:45.210391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.210298    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a1e1f1\""
Sep 09 12:24:45.216311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.216288    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:24:45.225680 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.225653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.225852 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.225758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55vz\" (UniqueName: \"kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.254414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.254382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:24:45.257674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.257656    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.268913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.268886    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:24:45.327117 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.327074    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgflf\" (UniqueName: \"kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.327605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.327184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.327605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.327312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.327605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.327381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55vz\" (UniqueName: \"kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.328286 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.328259    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.336498 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.336471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55vz\" (UniqueName: \"kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz\") pod \"payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") " pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.427800 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.427757    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgflf\" (UniqueName: \"kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.427974 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.427823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.428720 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.428697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.437220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.437189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgflf\" (UniqueName: \"kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf\") pod \"payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.520003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.519970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:45.569257 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.569203    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:45.797851 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.797826    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:24:45.799674 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:45.799645    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9f2a02_5cf7_4544_b4dd_c57a77bd1686.slice/crio-45483943516a8de32934226be3e9767504212ee9daf509ac3dd1904437dbfe8c WatchSource:0}: Error finding container 45483943516a8de32934226be3e9767504212ee9daf509ac3dd1904437dbfe8c: Status 404 returned error can't find the container with id 45483943516a8de32934226be3e9767504212ee9daf509ac3dd1904437dbfe8c
Sep 09 12:24:45.861316 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:45.861284    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:24:45.862720 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:24:45.862696    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c84a57_7501_42b1_adf0_15784d3a2b46.slice/crio-bcb7d4870abd223a37a571badf6ef3bacba66331d4e3b245e9b8dce09067874d WatchSource:0}: Error finding container bcb7d4870abd223a37a571badf6ef3bacba66331d4e3b245e9b8dce09067874d: Status 404 returned error can't find the container with id bcb7d4870abd223a37a571badf6ef3bacba66331d4e3b245e9b8dce09067874d
Sep 09 12:24:46.231932 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.231902    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:24:46.297006 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.296967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" event={"ID":"58c84a57-7501-42b1-adf0-15784d3a2b46","Type":"ContainerStarted","Data":"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"}
Sep 09 12:24:46.297220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.297032    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" event={"ID":"58c84a57-7501-42b1-adf0-15784d3a2b46","Type":"ContainerStarted","Data":"bcb7d4870abd223a37a571badf6ef3bacba66331d4e3b245e9b8dce09067874d"}
Sep 09 12:24:46.297220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.297078    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:46.299763 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.299731    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" event={"ID":"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686","Type":"ContainerStarted","Data":"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"}
Sep 09 12:24:46.299886 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.299794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" event={"ID":"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686","Type":"ContainerStarted","Data":"45483943516a8de32934226be3e9767504212ee9daf509ac3dd1904437dbfe8c"}
Sep 09 12:24:46.300268 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.300233    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:24:46.314431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.314370    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" podStartSLOduration=1.314349461 podStartE2EDuration="1.314349461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:46.31365338 +0000 UTC m=+808.944241155" watchObservedRunningTime="2026-09-09 12:24:46.314349461 +0000 UTC m=+808.944937237"
Sep 09 12:24:46.331612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:46.331565    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" podStartSLOduration=1.331551449 podStartE2EDuration="1.331551449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:46.330551655 +0000 UTC m=+808.961139423" watchObservedRunningTime="2026-09-09 12:24:46.331551449 +0000 UTC m=+808.962139253"
Sep 09 12:24:47.241588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:47.241557    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:24:48.244994 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:48.244969    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:24:48.245423 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:48.245029    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:24:55.289446 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:55.289414    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:24:55.289838 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:55.289645    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:24:57.307688 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:57.307652    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:24:57.308148 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:24:57.307729    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:25:02.572664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.572627    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:25:02.576250 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.576225    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.578484 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.578453    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a142a\""
Sep 09 12:25:02.586727 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.586701    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:25:02.660037 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.659996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.660222 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.660061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7w2\" (UniqueName: \"kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.761451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.761405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7w2\" (UniqueName: \"kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.761629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.761511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.762219 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.762197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.770927 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.770901    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7w2\" (UniqueName: \"kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2\") pod \"payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:02.888157 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:02.887927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:03.172096 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:03.172072    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:25:03.174325 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:25:03.174291    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31a81d0_0aa4_43cb_9fda_cc80d114c936.slice/crio-318e5c2d41f5eea916c6b0a04e97d0bc739260fcfa4a34e8b6ffd033a5ff8e9d WatchSource:0}: Error finding container 318e5c2d41f5eea916c6b0a04e97d0bc739260fcfa4a34e8b6ffd033a5ff8e9d: Status 404 returned error can't find the container with id 318e5c2d41f5eea916c6b0a04e97d0bc739260fcfa4a34e8b6ffd033a5ff8e9d
Sep 09 12:25:03.394895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:03.394863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" event={"ID":"c31a81d0-0aa4-43cb-9fda-cc80d114c936","Type":"ContainerStarted","Data":"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"}
Sep 09 12:25:03.395085 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:03.394914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" event={"ID":"c31a81d0-0aa4-43cb-9fda-cc80d114c936","Type":"ContainerStarted","Data":"318e5c2d41f5eea916c6b0a04e97d0bc739260fcfa4a34e8b6ffd033a5ff8e9d"}
Sep 09 12:25:03.395085 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:03.394955    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:03.413426 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:03.413371    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" podStartSLOduration=1.413329004 podStartE2EDuration="1.413329004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:25:03.41250226 +0000 UTC m=+826.043090035" watchObservedRunningTime="2026-09-09 12:25:03.413329004 +0000 UTC m=+826.043916780"
Sep 09 12:25:14.401173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:14.401140    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:25:24.219928 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.219892    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:25:24.220352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.220119    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" podUID="43bfcc67-320b-4aff-8f05-2768033287a3" containerName="payload-processing" containerID="cri-o://8ef3488b0dc7e18178bfbb788379d7439ed264109fbc0ac5086efd88311437a0" gracePeriod=30
Sep 09 12:25:24.231070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.231038    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:25:24.231334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.231308    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" podUID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" containerName="payload-pre-processing" containerID="cri-o://5cb827631466e527363515402c7c04ad3eb449c1788eaf4a91ee5a49f49c0a02" gracePeriod=30
Sep 09 12:25:24.496394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.496108    2556 generic.go:356] "Generic (PLEG): container finished" podID="43bfcc67-320b-4aff-8f05-2768033287a3" containerID="8ef3488b0dc7e18178bfbb788379d7439ed264109fbc0ac5086efd88311437a0" exitCode=0
Sep 09 12:25:24.496394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.496273    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" event={"ID":"43bfcc67-320b-4aff-8f05-2768033287a3","Type":"ContainerDied","Data":"8ef3488b0dc7e18178bfbb788379d7439ed264109fbc0ac5086efd88311437a0"}
Sep 09 12:25:24.498682 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.498655    2556 generic.go:356] "Generic (PLEG): container finished" podID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" containerID="5cb827631466e527363515402c7c04ad3eb449c1788eaf4a91ee5a49f49c0a02" exitCode=0
Sep 09 12:25:24.498800 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.498708    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" event={"ID":"66ae97e6-28a7-42bb-86b8-94584ff8ca64","Type":"ContainerDied","Data":"5cb827631466e527363515402c7c04ad3eb449c1788eaf4a91ee5a49f49c0a02"}
Sep 09 12:25:24.521911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.521885    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:25:24.524619 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.524600    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:25:24.664497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.664461    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume\") pod \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") "
Sep 09 12:25:24.664497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.664507    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jjkj\" (UniqueName: \"kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj\") pod \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\" (UID: \"66ae97e6-28a7-42bb-86b8-94584ff8ca64\") "
Sep 09 12:25:24.664729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.664703    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gts6k\" (UniqueName: \"kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k\") pod \"43bfcc67-320b-4aff-8f05-2768033287a3\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") "
Sep 09 12:25:24.664793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.664779    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume\") pod \"43bfcc67-320b-4aff-8f05-2768033287a3\" (UID: \"43bfcc67-320b-4aff-8f05-2768033287a3\") "
Sep 09 12:25:24.664884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.664841    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66ae97e6-28a7-42bb-86b8-94584ff8ca64" (UID: "66ae97e6-28a7-42bb-86b8-94584ff8ca64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:25:24.665083 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.665050    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ae97e6-28a7-42bb-86b8-94584ff8ca64-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:24.665187 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.665158    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43bfcc67-320b-4aff-8f05-2768033287a3" (UID: "43bfcc67-320b-4aff-8f05-2768033287a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:25:24.666968 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.666944    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj" (OuterVolumeSpecName: "kube-api-access-5jjkj") pod "66ae97e6-28a7-42bb-86b8-94584ff8ca64" (UID: "66ae97e6-28a7-42bb-86b8-94584ff8ca64"). InnerVolumeSpecName "kube-api-access-5jjkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:25:24.667153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.667136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k" (OuterVolumeSpecName: "kube-api-access-gts6k") pod "43bfcc67-320b-4aff-8f05-2768033287a3" (UID: "43bfcc67-320b-4aff-8f05-2768033287a3"). InnerVolumeSpecName "kube-api-access-gts6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:25:24.765884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.765786    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bfcc67-320b-4aff-8f05-2768033287a3-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:24.765884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.765822    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jjkj\" (UniqueName: \"kubernetes.io/projected/66ae97e6-28a7-42bb-86b8-94584ff8ca64-kube-api-access-5jjkj\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:24.765884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:24.765835    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gts6k\" (UniqueName: \"kubernetes.io/projected/43bfcc67-320b-4aff-8f05-2768033287a3-kube-api-access-gts6k\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:25.505225 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.505197    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"
Sep 09 12:25:25.505713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.505252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd" event={"ID":"66ae97e6-28a7-42bb-86b8-94584ff8ca64","Type":"ContainerDied","Data":"be3599d4a3689878444af074578a9cca5ea6d4322967482b9c6ad9a91a075caa"}
Sep 09 12:25:25.505713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.505333    2556 scope.go:117] "RemoveContainer" containerID="5cb827631466e527363515402c7c04ad3eb449c1788eaf4a91ee5a49f49c0a02"
Sep 09 12:25:25.506917 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.506897    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"
Sep 09 12:25:25.507017 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.506929    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds" event={"ID":"43bfcc67-320b-4aff-8f05-2768033287a3","Type":"ContainerDied","Data":"b7ca6e3130ac3837a8495dcf1972281abf32d47ae94503e6a88c0c6e05090c8c"}
Sep 09 12:25:25.514478 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.514460    2556 scope.go:117] "RemoveContainer" containerID="8ef3488b0dc7e18178bfbb788379d7439ed264109fbc0ac5086efd88311437a0"
Sep 09 12:25:25.537479 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.537451    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:25:25.540208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.540181    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c801b865-bdf9bf8b-pjgmd"]
Sep 09 12:25:25.557293 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.557260    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:25:25.561815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.561790    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c801b865-5554f78b9c-grtds"]
Sep 09 12:25:25.907025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.906940    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43bfcc67-320b-4aff-8f05-2768033287a3" path="/var/lib/kubelet/pods/43bfcc67-320b-4aff-8f05-2768033287a3/volumes"
Sep 09 12:25:25.907423 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:25.907403    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" path="/var/lib/kubelet/pods/66ae97e6-28a7-42bb-86b8-94584ff8ca64/volumes"
Sep 09 12:25:26.585950 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.585914    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:25:26.586363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586292    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" containerName="payload-pre-processing"
Sep 09 12:25:26.586363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586306    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" containerName="payload-pre-processing"
Sep 09 12:25:26.586363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586324    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43bfcc67-320b-4aff-8f05-2768033287a3" containerName="payload-processing"
Sep 09 12:25:26.586363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586332    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="43bfcc67-320b-4aff-8f05-2768033287a3" containerName="payload-processing"
Sep 09 12:25:26.586537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586412    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="43bfcc67-320b-4aff-8f05-2768033287a3" containerName="payload-processing"
Sep 09 12:25:26.586537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.586419    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ae97e6-28a7-42bb-86b8-94584ff8ca64" containerName="payload-pre-processing"
Sep 09 12:25:26.590889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.590871    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.594635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.594608    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c554db\"/\"default-dockercfg-lwttb\""
Sep 09 12:25:26.594791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.594637    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c554db\"/\"openshift-service-ca.crt\""
Sep 09 12:25:26.594791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.594638    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c554db\"/\"simulator-w0-c554db-kserve-self-signed-certs\""
Sep 09 12:25:26.594791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.594700    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c554db\"/\"kube-root-ca.crt\""
Sep 09 12:25:26.601638 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.601604    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:25:26.685915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.685816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.685915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.685897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrbn\" (UniqueName: \"kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.686180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.685931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.686180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.685978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.686180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.686006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.686180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.686055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787566 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787777 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787597    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrbn\" (UniqueName: \"kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787777 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787616    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787777 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787777 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787677    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.787777 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.787724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.788034 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.788016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.788108 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.788089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.788159 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.788125    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.790143 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.790119    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.790345 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.790325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.796279 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.796225    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrbn\" (UniqueName: \"kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn\") pod \"simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") " pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:26.900336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:26.900222    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:27.176309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:27.176228    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:25:27.178855 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:25:27.178829    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd99e616_1fa6_42aa_8568_e1460a3361a9.slice/crio-109b5f2882fbb897d8bfd1e0c40a00d02f316ab056e87f83edaf6563d954a093 WatchSource:0}: Error finding container 109b5f2882fbb897d8bfd1e0c40a00d02f316ab056e87f83edaf6563d954a093: Status 404 returned error can't find the container with id 109b5f2882fbb897d8bfd1e0c40a00d02f316ab056e87f83edaf6563d954a093
Sep 09 12:25:27.520231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:27.520195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" event={"ID":"cd99e616-1fa6-42aa-8568-e1460a3361a9","Type":"ContainerStarted","Data":"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"}
Sep 09 12:25:27.520231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:27.520258    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" event={"ID":"cd99e616-1fa6-42aa-8568-e1460a3361a9","Type":"ContainerStarted","Data":"109b5f2882fbb897d8bfd1e0c40a00d02f316ab056e87f83edaf6563d954a093"}
Sep 09 12:25:27.539125 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:27.539072    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" podStartSLOduration=1.539055463 podStartE2EDuration="1.539055463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:25:27.538126408 +0000 UTC m=+850.168714185" watchObservedRunningTime="2026-09-09 12:25:27.539055463 +0000 UTC m=+850.169643239"
Sep 09 12:25:29.188362 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.188324    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:29.191849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.191827    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.194254 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.194208    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-669203b2-dockercfg-lwxjk\""
Sep 09 12:25:29.194385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.194262    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-669203b2\""
Sep 09 12:25:29.203717 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.203691    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:29.212986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.212956    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:29.216542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.216522    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.227609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.227585    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:29.311849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.311807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.311849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.311857    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.312066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.311920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc57n\" (UniqueName: \"kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.312066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.311961    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz29j\" (UniqueName: \"kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.413170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.413420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413189    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc57n\" (UniqueName: \"kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.413420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz29j\" (UniqueName: \"kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.413420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.413934 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413914    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.413984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.413945    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.422412 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.422389    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc57n\" (UniqueName: \"kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n\") pod \"payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.422698 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.422681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz29j\" (UniqueName: \"kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j\") pod \"payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") " pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.503835 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.503795    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:29.526587 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.526554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:29.779614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.779584    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:29.782183 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:25:29.782154    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c6965c_c4b1_4cd7_92d0_03a9eeee5bf0.slice/crio-3ad28770f869e2c31fe77b243bfe874f617461a362f472858470842431ef0be3 WatchSource:0}: Error finding container 3ad28770f869e2c31fe77b243bfe874f617461a362f472858470842431ef0be3: Status 404 returned error can't find the container with id 3ad28770f869e2c31fe77b243bfe874f617461a362f472858470842431ef0be3
Sep 09 12:25:29.810939 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:29.810909    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:29.816711 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:25:29.816683    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848a3983_bbf9_415e_ab76_07e955b885d1.slice/crio-5a11b80cab43b6304bc58df3b8251955245d5d64e60af4f9aa7fd20ef03f3001 WatchSource:0}: Error finding container 5a11b80cab43b6304bc58df3b8251955245d5d64e60af4f9aa7fd20ef03f3001: Status 404 returned error can't find the container with id 5a11b80cab43b6304bc58df3b8251955245d5d64e60af4f9aa7fd20ef03f3001
Sep 09 12:25:30.536708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.536670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" event={"ID":"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0","Type":"ContainerStarted","Data":"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"}
Sep 09 12:25:30.537170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.536727    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" event={"ID":"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0","Type":"ContainerStarted","Data":"3ad28770f869e2c31fe77b243bfe874f617461a362f472858470842431ef0be3"}
Sep 09 12:25:30.537170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.536790    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:30.538473 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.538446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" event={"ID":"848a3983-bbf9-415e-ab76-07e955b885d1","Type":"ContainerStarted","Data":"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"}
Sep 09 12:25:30.538544 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.538496    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" event={"ID":"848a3983-bbf9-415e-ab76-07e955b885d1","Type":"ContainerStarted","Data":"5a11b80cab43b6304bc58df3b8251955245d5d64e60af4f9aa7fd20ef03f3001"}
Sep 09 12:25:30.538544 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.538525    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:30.553700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.553655    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" podStartSLOduration=1.55363999 podStartE2EDuration="1.55363999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:25:30.552336426 +0000 UTC m=+853.182924202" watchObservedRunningTime="2026-09-09 12:25:30.55363999 +0000 UTC m=+853.184227765"
Sep 09 12:25:30.569561 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:30.569507    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" podStartSLOduration=1.569490169 podStartE2EDuration="1.569490169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:25:30.567337468 +0000 UTC m=+853.197925237" watchObservedRunningTime="2026-09-09 12:25:30.569490169 +0000 UTC m=+853.200077944"
Sep 09 12:25:36.900648 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:36.900612    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:36.900648 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:36.900655    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:36.912955 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:36.912928    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:37.586283 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:37.586230    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:25:41.546436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:41.546396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:41.546808 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:41.546470    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:59.192292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.192251    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:59.192710 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.192487    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" podUID="848a3983-bbf9-415e-ab76-07e955b885d1" containerName="payload-processing" containerID="cri-o://6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9" gracePeriod=30
Sep 09 12:25:59.199468 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.199444    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:59.199684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.199652    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" podUID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" containerName="payload-pre-processing" containerID="cri-o://534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893" gracePeriod=30
Sep 09 12:25:59.483104 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.483075    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:59.487070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.487046    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:59.588180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588139    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc57n\" (UniqueName: \"kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n\") pod \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") "
Sep 09 12:25:59.588376 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588202    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume\") pod \"848a3983-bbf9-415e-ab76-07e955b885d1\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") "
Sep 09 12:25:59.588376 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588220    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz29j\" (UniqueName: \"kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j\") pod \"848a3983-bbf9-415e-ab76-07e955b885d1\" (UID: \"848a3983-bbf9-415e-ab76-07e955b885d1\") "
Sep 09 12:25:59.588376 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588273    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume\") pod \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\" (UID: \"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0\") "
Sep 09 12:25:59.588595 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588571    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "848a3983-bbf9-415e-ab76-07e955b885d1" (UID: "848a3983-bbf9-415e-ab76-07e955b885d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:25:59.588706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.588678    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" (UID: "f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:25:59.590652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.590627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n" (OuterVolumeSpecName: "kube-api-access-mc57n") pod "f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" (UID: "f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0"). InnerVolumeSpecName "kube-api-access-mc57n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:25:59.590751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.590652    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j" (OuterVolumeSpecName: "kube-api-access-sz29j") pod "848a3983-bbf9-415e-ab76-07e955b885d1" (UID: "848a3983-bbf9-415e-ab76-07e955b885d1"). InnerVolumeSpecName "kube-api-access-sz29j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:25:59.689060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689026    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc57n\" (UniqueName: \"kubernetes.io/projected/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-kube-api-access-mc57n\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:59.689060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689058    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848a3983-bbf9-415e-ab76-07e955b885d1-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:59.689340 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689082    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz29j\" (UniqueName: \"kubernetes.io/projected/848a3983-bbf9-415e-ab76-07e955b885d1-kube-api-access-sz29j\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:59.689340 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689091    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:25:59.689629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689604    2556 generic.go:356] "Generic (PLEG): container finished" podID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" containerID="534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893" exitCode=0
Sep 09 12:25:59.689671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689660    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"
Sep 09 12:25:59.689727 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" event={"ID":"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0","Type":"ContainerDied","Data":"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"}
Sep 09 12:25:59.689788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks" event={"ID":"f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0","Type":"ContainerDied","Data":"3ad28770f869e2c31fe77b243bfe874f617461a362f472858470842431ef0be3"}
Sep 09 12:25:59.689828 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.689807    2556 scope.go:117] "RemoveContainer" containerID="534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"
Sep 09 12:25:59.691101 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.691074    2556 generic.go:356] "Generic (PLEG): container finished" podID="848a3983-bbf9-415e-ab76-07e955b885d1" containerID="6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9" exitCode=0
Sep 09 12:25:59.691195 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.691112    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"
Sep 09 12:25:59.691195 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.691155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" event={"ID":"848a3983-bbf9-415e-ab76-07e955b885d1","Type":"ContainerDied","Data":"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"}
Sep 09 12:25:59.691312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.691198    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68" event={"ID":"848a3983-bbf9-415e-ab76-07e955b885d1","Type":"ContainerDied","Data":"5a11b80cab43b6304bc58df3b8251955245d5d64e60af4f9aa7fd20ef03f3001"}
Sep 09 12:25:59.699520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.699502    2556 scope.go:117] "RemoveContainer" containerID="534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"
Sep 09 12:25:59.699894 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:25:59.699872    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893\": container with ID starting with 534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893 not found: ID does not exist" containerID="534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"
Sep 09 12:25:59.699964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.699905    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893"} err="failed to get container status \"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893\": rpc error: code = NotFound desc = could not find container \"534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893\": container with ID starting with 534d5765a89dc23c6c2a78850eb606037d117847d5c69b2d161351ecc63ab893 not found: ID does not exist"
Sep 09 12:25:59.699964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.699928    2556 scope.go:117] "RemoveContainer" containerID="6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"
Sep 09 12:25:59.708551 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.708526    2556 scope.go:117] "RemoveContainer" containerID="6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"
Sep 09 12:25:59.708868 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:25:59.708847    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9\": container with ID starting with 6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9 not found: ID does not exist" containerID="6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"
Sep 09 12:25:59.708935 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.708881    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9"} err="failed to get container status \"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9\": rpc error: code = NotFound desc = could not find container \"6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9\": container with ID starting with 6d4cc8f5bb1731147ecd808df03bd6756fbd0faf2d19920f48b048ad5e5f79c9 not found: ID does not exist"
Sep 09 12:25:59.717336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.717300    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:59.722285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.722255    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-669203b2-77f96d9b54-94pks"]
Sep 09 12:25:59.734038 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.733999    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:59.738131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.738106    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-669203b2-7c4f446654-xrg68"]
Sep 09 12:25:59.902052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.902018    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848a3983-bbf9-415e-ab76-07e955b885d1" path="/var/lib/kubelet/pods/848a3983-bbf9-415e-ab76-07e955b885d1/volumes"
Sep 09 12:25:59.902401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:25:59.902387    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" path="/var/lib/kubelet/pods/f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0/volumes"
Sep 09 12:26:04.436511 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436479    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:04.436880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436841    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848a3983-bbf9-415e-ab76-07e955b885d1" containerName="payload-processing"
Sep 09 12:26:04.436880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436852    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="848a3983-bbf9-415e-ab76-07e955b885d1" containerName="payload-processing"
Sep 09 12:26:04.436880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436864    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" containerName="payload-pre-processing"
Sep 09 12:26:04.436880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436869    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" containerName="payload-pre-processing"
Sep 09 12:26:04.437007 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436933    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c6965c-c4b1-4cd7-92d0-03a9eeee5bf0" containerName="payload-pre-processing"
Sep 09 12:26:04.437007 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.436943    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="848a3983-bbf9-415e-ab76-07e955b885d1" containerName="payload-processing"
Sep 09 12:26:04.441285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.441265    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.443644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.443621    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ee5335d\""
Sep 09 12:26:04.450014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.449991    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:04.479416 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.479388    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:04.482634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.482615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.485104 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.485077    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ee5335d-dockercfg-rnfh4\""
Sep 09 12:26:04.492099 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.492071    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:04.533180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.533146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckg2\" (UniqueName: \"kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.533378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.533196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn75p\" (UniqueName: \"kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.533378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.533314    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.533378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.533342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.634207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.634163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckg2\" (UniqueName: \"kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.634207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.634216    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn75p\" (UniqueName: \"kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.634509 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.634284    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.634509 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.634306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.635208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.635184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.635272 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.635184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.643501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.643472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckg2\" (UniqueName: \"kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2\") pod \"payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.643638 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.643477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn75p\" (UniqueName: \"kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p\") pod \"payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") " pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:04.780677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.780630    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:04.794265 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:04.794217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:05.061267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.061123    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:05.068580 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:26:05.068530    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead0400d_ea63_4941_8996_589f9b6b073d.slice/crio-79034a6d3e84722625a3c4673d9f2fcbcc4de9d61690392ee5001c6b06e62ae1 WatchSource:0}: Error finding container 79034a6d3e84722625a3c4673d9f2fcbcc4de9d61690392ee5001c6b06e62ae1: Status 404 returned error can't find the container with id 79034a6d3e84722625a3c4673d9f2fcbcc4de9d61690392ee5001c6b06e62ae1
Sep 09 12:26:05.082744 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.082721    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:05.085211 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:26:05.085180    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39747cda_cfd8_4a80_99a0_27d92a792e0d.slice/crio-a8186f0ae1f68d7837fd9b502e625245ac3a83f683658b9892169c30ec7e607d WatchSource:0}: Error finding container a8186f0ae1f68d7837fd9b502e625245ac3a83f683658b9892169c30ec7e607d: Status 404 returned error can't find the container with id a8186f0ae1f68d7837fd9b502e625245ac3a83f683658b9892169c30ec7e607d
Sep 09 12:26:05.726555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.726517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" event={"ID":"ead0400d-ea63-4941-8996-589f9b6b073d","Type":"ContainerStarted","Data":"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"}
Sep 09 12:26:05.727030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.726599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" event={"ID":"ead0400d-ea63-4941-8996-589f9b6b073d","Type":"ContainerStarted","Data":"79034a6d3e84722625a3c4673d9f2fcbcc4de9d61690392ee5001c6b06e62ae1"}
Sep 09 12:26:05.727030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.726647    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:05.728491 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.728466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" event={"ID":"39747cda-cfd8-4a80-99a0-27d92a792e0d","Type":"ContainerStarted","Data":"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"}
Sep 09 12:26:05.728604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.728514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" event={"ID":"39747cda-cfd8-4a80-99a0-27d92a792e0d","Type":"ContainerStarted","Data":"a8186f0ae1f68d7837fd9b502e625245ac3a83f683658b9892169c30ec7e607d"}
Sep 09 12:26:05.728604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.728549    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:05.744760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.744699    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" podStartSLOduration=1.74467799 podStartE2EDuration="1.74467799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:05.741742128 +0000 UTC m=+888.372329903" watchObservedRunningTime="2026-09-09 12:26:05.74467799 +0000 UTC m=+888.375265768"
Sep 09 12:26:05.758618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:05.758560    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" podStartSLOduration=1.75854299 podStartE2EDuration="1.75854299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:05.757371071 +0000 UTC m=+888.387958847" watchObservedRunningTime="2026-09-09 12:26:05.75854299 +0000 UTC m=+888.389130765"
Sep 09 12:26:16.736154 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:16.736074    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:16.736756 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:16.736283    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:17.820405 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:17.820378    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:26:17.821938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:17.821919    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:26:17.824372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:17.824356    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:26:17.825847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:17.825831    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:26:23.390760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.390725    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:26:23.395641 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.395618    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.398065 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.398038    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a1e1f1\"/\"openshift-service-ca.crt\""
Sep 09 12:26:23.398186 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.398089    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a1e1f1\"/\"kube-root-ca.crt\""
Sep 09 12:26:23.398992 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.398976    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a1e1f1\"/\"autores-model-a1e1f1-kserve-self-signed-certs\""
Sep 09 12:26:23.399100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.399014    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a1e1f1\"/\"default-dockercfg-hvtft\""
Sep 09 12:26:23.404670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.404647    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:26:23.507517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgjm\" (UniqueName: \"kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.507517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.507783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.507783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.507783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.507783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.507750    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609093 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgjm\" (UniqueName: \"kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.609731 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.609712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.611752 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.611724    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.611936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.611921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.618516 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.618488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgjm\" (UniqueName: \"kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm\") pod \"autores-model-a1e1f1-kserve-c78cd65dc-2zt59\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") " pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.707431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.707329    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:23.984609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:23.984575    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:26:23.986937 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:26:23.986904    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd69c41a_2af3_4665_b1a6_28f6a73119fc.slice/crio-5da72ed23332872b86d7a724c9a10e5bb19b4eb1da2324156fffd15e8c5da638 WatchSource:0}: Error finding container 5da72ed23332872b86d7a724c9a10e5bb19b4eb1da2324156fffd15e8c5da638: Status 404 returned error can't find the container with id 5da72ed23332872b86d7a724c9a10e5bb19b4eb1da2324156fffd15e8c5da638
Sep 09 12:26:24.823225 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:24.823187    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" event={"ID":"cd69c41a-2af3-4665-b1a6-28f6a73119fc","Type":"ContainerStarted","Data":"6572b64038f5a8f8d374d4b046f23e065a48b4ad5c33c9f0269930e5819042b4"}
Sep 09 12:26:24.823632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:24.823230    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" event={"ID":"cd69c41a-2af3-4665-b1a6-28f6a73119fc","Type":"ContainerStarted","Data":"5da72ed23332872b86d7a724c9a10e5bb19b4eb1da2324156fffd15e8c5da638"}
Sep 09 12:26:24.842801 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:24.842740    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" podStartSLOduration=1.842719327 podStartE2EDuration="1.842719327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:24.839775395 +0000 UTC m=+907.470363170" watchObservedRunningTime="2026-09-09 12:26:24.842719327 +0000 UTC m=+907.473307104"
Sep 09 12:26:33.707926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:33.707884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:33.708407 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:33.707945    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:33.722514 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:33.722483    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:33.878883 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:33.878844    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:26:34.388311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.388272    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:34.388562 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.388534    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" podUID="39747cda-cfd8-4a80-99a0-27d92a792e0d" containerName="payload-processing" containerID="cri-o://e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f" gracePeriod=30
Sep 09 12:26:34.394597 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.394566    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:34.394860 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.394804    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" podUID="ead0400d-ea63-4941-8996-589f9b6b073d" containerName="payload-pre-processing" containerID="cri-o://4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1" gracePeriod=30
Sep 09 12:26:34.662120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.662083    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:34.679924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.679897    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:34.712620 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.712581    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qckg2\" (UniqueName: \"kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2\") pod \"ead0400d-ea63-4941-8996-589f9b6b073d\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") "
Sep 09 12:26:34.713088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.712695    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume\") pod \"ead0400d-ea63-4941-8996-589f9b6b073d\" (UID: \"ead0400d-ea63-4941-8996-589f9b6b073d\") "
Sep 09 12:26:34.713088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.712717    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume\") pod \"39747cda-cfd8-4a80-99a0-27d92a792e0d\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") "
Sep 09 12:26:34.713088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.712769    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn75p\" (UniqueName: \"kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p\") pod \"39747cda-cfd8-4a80-99a0-27d92a792e0d\" (UID: \"39747cda-cfd8-4a80-99a0-27d92a792e0d\") "
Sep 09 12:26:34.713292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.713143    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39747cda-cfd8-4a80-99a0-27d92a792e0d" (UID: "39747cda-cfd8-4a80-99a0-27d92a792e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:26:34.713292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.713156    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ead0400d-ea63-4941-8996-589f9b6b073d" (UID: "ead0400d-ea63-4941-8996-589f9b6b073d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:26:34.715016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.714987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p" (OuterVolumeSpecName: "kube-api-access-xn75p") pod "39747cda-cfd8-4a80-99a0-27d92a792e0d" (UID: "39747cda-cfd8-4a80-99a0-27d92a792e0d"). InnerVolumeSpecName "kube-api-access-xn75p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:26:34.715016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.715011    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2" (OuterVolumeSpecName: "kube-api-access-qckg2") pod "ead0400d-ea63-4941-8996-589f9b6b073d" (UID: "ead0400d-ea63-4941-8996-589f9b6b073d"). InnerVolumeSpecName "kube-api-access-qckg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:26:34.813853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.813814    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn75p\" (UniqueName: \"kubernetes.io/projected/39747cda-cfd8-4a80-99a0-27d92a792e0d-kube-api-access-xn75p\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:26:34.813853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.813846    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckg2\" (UniqueName: \"kubernetes.io/projected/ead0400d-ea63-4941-8996-589f9b6b073d-kube-api-access-qckg2\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:26:34.813853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.813855    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead0400d-ea63-4941-8996-589f9b6b073d-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:26:34.814090 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.813863    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39747cda-cfd8-4a80-99a0-27d92a792e0d-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:26:34.874542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.874505    2556 generic.go:356] "Generic (PLEG): container finished" podID="ead0400d-ea63-4941-8996-589f9b6b073d" containerID="4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1" exitCode=0
Sep 09 12:26:34.874677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.874554    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"
Sep 09 12:26:34.874677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.874592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" event={"ID":"ead0400d-ea63-4941-8996-589f9b6b073d","Type":"ContainerDied","Data":"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"}
Sep 09 12:26:34.874677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.874650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb" event={"ID":"ead0400d-ea63-4941-8996-589f9b6b073d","Type":"ContainerDied","Data":"79034a6d3e84722625a3c4673d9f2fcbcc4de9d61690392ee5001c6b06e62ae1"}
Sep 09 12:26:34.874677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.874675    2556 scope.go:117] "RemoveContainer" containerID="4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"
Sep 09 12:26:34.875912 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.875884    2556 generic.go:356] "Generic (PLEG): container finished" podID="39747cda-cfd8-4a80-99a0-27d92a792e0d" containerID="e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f" exitCode=0
Sep 09 12:26:34.876030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.875932    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"
Sep 09 12:26:34.876030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.875958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" event={"ID":"39747cda-cfd8-4a80-99a0-27d92a792e0d","Type":"ContainerDied","Data":"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"}
Sep 09 12:26:34.876030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.876005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k" event={"ID":"39747cda-cfd8-4a80-99a0-27d92a792e0d","Type":"ContainerDied","Data":"a8186f0ae1f68d7837fd9b502e625245ac3a83f683658b9892169c30ec7e607d"}
Sep 09 12:26:34.883987 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.883929    2556 scope.go:117] "RemoveContainer" containerID="4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"
Sep 09 12:26:34.884294 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:26:34.884273    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1\": container with ID starting with 4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1 not found: ID does not exist" containerID="4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"
Sep 09 12:26:34.884349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.884307    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1"} err="failed to get container status \"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1\": rpc error: code = NotFound desc = could not find container \"4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1\": container with ID starting with 4f8f94b6e4736bdec0d92a9e2d2d7bf0e62f478ca65611cf14c77baf1d2b5bb1 not found: ID does not exist"
Sep 09 12:26:34.884349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.884329    2556 scope.go:117] "RemoveContainer" containerID="e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"
Sep 09 12:26:34.892792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.892773    2556 scope.go:117] "RemoveContainer" containerID="e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"
Sep 09 12:26:34.893096 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:26:34.893059    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f\": container with ID starting with e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f not found: ID does not exist" containerID="e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"
Sep 09 12:26:34.893164 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.893101    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f"} err="failed to get container status \"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f\": rpc error: code = NotFound desc = could not find container \"e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f\": container with ID starting with e5db3524b84b19618a6bfcfdca26806859fc27996aab890188a9b624ad00682f not found: ID does not exist"
Sep 09 12:26:34.901380 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.901353    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:34.906064 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.906031    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ee5335d-5c5558c8ff-z2s4k"]
Sep 09 12:26:34.917424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.917392    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:34.922085 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:34.922058    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ee5335d-fc7c8f9bb-xggcb"]
Sep 09 12:26:35.902306 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:35.902272    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39747cda-cfd8-4a80-99a0-27d92a792e0d" path="/var/lib/kubelet/pods/39747cda-cfd8-4a80-99a0-27d92a792e0d/volumes"
Sep 09 12:26:35.902690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:35.902641    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead0400d-ea63-4941-8996-589f9b6b073d" path="/var/lib/kubelet/pods/ead0400d-ea63-4941-8996-589f9b6b073d/volumes"
Sep 09 12:26:40.414028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.413985    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:26:40.414438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414388    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead0400d-ea63-4941-8996-589f9b6b073d" containerName="payload-pre-processing"
Sep 09 12:26:40.414438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414401    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead0400d-ea63-4941-8996-589f9b6b073d" containerName="payload-pre-processing"
Sep 09 12:26:40.414438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414426    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39747cda-cfd8-4a80-99a0-27d92a792e0d" containerName="payload-processing"
Sep 09 12:26:40.414438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414432    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="39747cda-cfd8-4a80-99a0-27d92a792e0d" containerName="payload-processing"
Sep 09 12:26:40.414567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414505    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead0400d-ea63-4941-8996-589f9b6b073d" containerName="payload-pre-processing"
Sep 09 12:26:40.414567 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.414516    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="39747cda-cfd8-4a80-99a0-27d92a792e0d" containerName="payload-processing"
Sep 09 12:26:40.419194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.419167    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.421972 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.421942    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e3e6ebae\""
Sep 09 12:26:40.427462 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.427437    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:26:40.445216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.445178    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:26:40.449040 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.449016    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.451965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.451934    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e3e6ebae-dockercfg-qjsrn\""
Sep 09 12:26:40.458594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.458565    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:26:40.464898 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.464867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.465059 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.464916    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlcpf\" (UniqueName: \"kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.566183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.566143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpxq\" (UniqueName: \"kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.566183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.566189    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.566421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.566302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlcpf\" (UniqueName: \"kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.566421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.566335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.566871 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.566853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.578656 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.578630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlcpf\" (UniqueName: \"kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf\") pod \"payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.667909 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.667810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpxq\" (UniqueName: \"kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.667909 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.667881    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.668607 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.668587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.677055 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.677024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpxq\" (UniqueName: \"kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq\") pod \"payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:40.731150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.731116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:40.760851 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:40.760815    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:41.017078 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.017052    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:26:41.018490 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:26:41.018460    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d1fbfd_b455_4a35_80bb_398439b3e990.slice/crio-25dfb876446906befd1d7567f3ab65a43d780848b0fe956629912f7a282d8ae0 WatchSource:0}: Error finding container 25dfb876446906befd1d7567f3ab65a43d780848b0fe956629912f7a282d8ae0: Status 404 returned error can't find the container with id 25dfb876446906befd1d7567f3ab65a43d780848b0fe956629912f7a282d8ae0
Sep 09 12:26:41.049562 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.049513    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:26:41.055407 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:26:41.055374    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8968fd_46b0_4c0d_8a1c_e722970afea1.slice/crio-b977d355f077e775d8e80a7087c19fff4584678a74ff6310f46af9fdc9d1f0a1 WatchSource:0}: Error finding container b977d355f077e775d8e80a7087c19fff4584678a74ff6310f46af9fdc9d1f0a1: Status 404 returned error can't find the container with id b977d355f077e775d8e80a7087c19fff4584678a74ff6310f46af9fdc9d1f0a1
Sep 09 12:26:41.912414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.912365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" event={"ID":"8b8968fd-46b0-4c0d-8a1c-e722970afea1","Type":"ContainerStarted","Data":"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"}
Sep 09 12:26:41.912977 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.912443    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" event={"ID":"8b8968fd-46b0-4c0d-8a1c-e722970afea1","Type":"ContainerStarted","Data":"b977d355f077e775d8e80a7087c19fff4584678a74ff6310f46af9fdc9d1f0a1"}
Sep 09 12:26:41.912977 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.912580    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:41.914109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.914080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" event={"ID":"e8d1fbfd-b455-4a35-80bb-398439b3e990","Type":"ContainerStarted","Data":"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"}
Sep 09 12:26:41.914265 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.914132    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:26:41.914265 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.914151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" event={"ID":"e8d1fbfd-b455-4a35-80bb-398439b3e990","Type":"ContainerStarted","Data":"25dfb876446906befd1d7567f3ab65a43d780848b0fe956629912f7a282d8ae0"}
Sep 09 12:26:41.935221 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.935164    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" podStartSLOduration=1.935147948 podStartE2EDuration="1.935147948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:41.933415255 +0000 UTC m=+924.564003032" watchObservedRunningTime="2026-09-09 12:26:41.935147948 +0000 UTC m=+924.565735723"
Sep 09 12:26:41.951469 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:41.951410    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" podStartSLOduration=1.9513928 podStartE2EDuration="1.9513928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:41.949397409 +0000 UTC m=+924.579985186" watchObservedRunningTime="2026-09-09 12:26:41.9513928 +0000 UTC m=+924.581980576"
Sep 09 12:26:52.924102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:52.924071    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:26:52.924555 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:26:52.924265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:27:02.875043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:02.875000    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:27:07.663251 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.663200    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:27:07.663732 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.663430    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" podUID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" containerName="payload-processing" containerID="cri-o://b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9" gracePeriod=30
Sep 09 12:27:07.671739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.671707    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:27:07.672449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.671950    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" podUID="58c84a57-7501-42b1-adf0-15784d3a2b46" containerName="payload-pre-processing" containerID="cri-o://83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8" gracePeriod=30
Sep 09 12:27:07.953956 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.953917    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:27:07.956833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:07.956812    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:27:08.032194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032164    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume\") pod \"58c84a57-7501-42b1-adf0-15784d3a2b46\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") "
Sep 09 12:27:08.032381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032208    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume\") pod \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") "
Sep 09 12:27:08.032381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032266    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b55vz\" (UniqueName: \"kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz\") pod \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\" (UID: \"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686\") "
Sep 09 12:27:08.032381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032285    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgflf\" (UniqueName: \"kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf\") pod \"58c84a57-7501-42b1-adf0-15784d3a2b46\" (UID: \"58c84a57-7501-42b1-adf0-15784d3a2b46\") "
Sep 09 12:27:08.032618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032596    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" (UID: "ea9f2a02-5cf7-4544-b4dd-c57a77bd1686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:27:08.032684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.032622    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58c84a57-7501-42b1-adf0-15784d3a2b46" (UID: "58c84a57-7501-42b1-adf0-15784d3a2b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:27:08.034519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.034496    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf" (OuterVolumeSpecName: "kube-api-access-dgflf") pod "58c84a57-7501-42b1-adf0-15784d3a2b46" (UID: "58c84a57-7501-42b1-adf0-15784d3a2b46"). InnerVolumeSpecName "kube-api-access-dgflf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:27:08.034640 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.034583    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz" (OuterVolumeSpecName: "kube-api-access-b55vz") pod "ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" (UID: "ea9f2a02-5cf7-4544-b4dd-c57a77bd1686"). InnerVolumeSpecName "kube-api-access-b55vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:27:08.054410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.054380    2556 generic.go:356] "Generic (PLEG): container finished" podID="58c84a57-7501-42b1-adf0-15784d3a2b46" containerID="83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8" exitCode=0
Sep 09 12:27:08.054539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.054437    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"
Sep 09 12:27:08.054539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.054446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" event={"ID":"58c84a57-7501-42b1-adf0-15784d3a2b46","Type":"ContainerDied","Data":"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"}
Sep 09 12:27:08.054539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.054518    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc" event={"ID":"58c84a57-7501-42b1-adf0-15784d3a2b46","Type":"ContainerDied","Data":"bcb7d4870abd223a37a571badf6ef3bacba66331d4e3b245e9b8dce09067874d"}
Sep 09 12:27:08.054690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.054548    2556 scope.go:117] "RemoveContainer" containerID="83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"
Sep 09 12:27:08.055831 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.055796    2556 generic.go:356] "Generic (PLEG): container finished" podID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" containerID="b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9" exitCode=0
Sep 09 12:27:08.055964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.055847    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"
Sep 09 12:27:08.055964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.055879    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" event={"ID":"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686","Type":"ContainerDied","Data":"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"}
Sep 09 12:27:08.055964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.055940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v" event={"ID":"ea9f2a02-5cf7-4544-b4dd-c57a77bd1686","Type":"ContainerDied","Data":"45483943516a8de32934226be3e9767504212ee9daf509ac3dd1904437dbfe8c"}
Sep 09 12:27:08.064993 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.064972    2556 scope.go:117] "RemoveContainer" containerID="83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"
Sep 09 12:27:08.065494 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:27:08.065263    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8\": container with ID starting with 83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8 not found: ID does not exist" containerID="83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"
Sep 09 12:27:08.065494 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.065295    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8"} err="failed to get container status \"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8\": rpc error: code = NotFound desc = could not find container \"83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8\": container with ID starting with 83c802a94acbe798cc6dc07e21a12ce195d35e017e5fcc4ababe139e9ff531d8 not found: ID does not exist"
Sep 09 12:27:08.065494 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.065322    2556 scope.go:117] "RemoveContainer" containerID="b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"
Sep 09 12:27:08.073801 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.073784    2556 scope.go:117] "RemoveContainer" containerID="b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"
Sep 09 12:27:08.074083 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:27:08.074066    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9\": container with ID starting with b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9 not found: ID does not exist" containerID="b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"
Sep 09 12:27:08.074122 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.074091    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9"} err="failed to get container status \"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9\": rpc error: code = NotFound desc = could not find container \"b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9\": container with ID starting with b1f350419001a1a668f4eec2c53686675512b102bbfb53bba41e549112fcbce9 not found: ID does not exist"
Sep 09 12:27:08.081367 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.081346    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:27:08.085693 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.085672    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1e1f1-7c68b8cbdb-6b9fc"]
Sep 09 12:27:08.096900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.096854    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:27:08.099195 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.099168    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1e1f1-84fbf489f-tp28v"]
Sep 09 12:27:08.133382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.133355    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:08.133382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.133380    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b55vz\" (UniqueName: \"kubernetes.io/projected/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686-kube-api-access-b55vz\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:08.133545 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.133392    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgflf\" (UniqueName: \"kubernetes.io/projected/58c84a57-7501-42b1-adf0-15784d3a2b46-kube-api-access-dgflf\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:08.133545 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:08.133401    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c84a57-7501-42b1-adf0-15784d3a2b46-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:09.902882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:09.902849    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c84a57-7501-42b1-adf0-15784d3a2b46" path="/var/lib/kubelet/pods/58c84a57-7501-42b1-adf0-15784d3a2b46/volumes"
Sep 09 12:27:09.903287 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:09.903204    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" path="/var/lib/kubelet/pods/ea9f2a02-5cf7-4544-b4dd-c57a77bd1686/volumes"
Sep 09 12:27:10.647736 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.647697    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:27:10.647979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.647949    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" podUID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" containerName="payload-processing" containerID="cri-o://e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6" gracePeriod=30
Sep 09 12:27:10.655913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.655815    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:27:10.656470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.656435    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" podUID="e8d1fbfd-b455-4a35-80bb-398439b3e990" containerName="payload-pre-processing" containerID="cri-o://c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63" gracePeriod=30
Sep 09 12:27:10.915876 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.915846    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:27:10.942418 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.942364    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:27:10.953406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953374    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume\") pod \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") "
Sep 09 12:27:10.953570 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953425    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume\") pod \"e8d1fbfd-b455-4a35-80bb-398439b3e990\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") "
Sep 09 12:27:10.953570 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953530    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlcpf\" (UniqueName: \"kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf\") pod \"e8d1fbfd-b455-4a35-80bb-398439b3e990\" (UID: \"e8d1fbfd-b455-4a35-80bb-398439b3e990\") "
Sep 09 12:27:10.953771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953749    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b8968fd-46b0-4c0d-8a1c-e722970afea1" (UID: "8b8968fd-46b0-4c0d-8a1c-e722970afea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:27:10.953854 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953813    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8d1fbfd-b455-4a35-80bb-398439b3e990" (UID: "e8d1fbfd-b455-4a35-80bb-398439b3e990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:27:10.953854 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.953830    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8968fd-46b0-4c0d-8a1c-e722970afea1-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:10.955867 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:10.955848    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf" (OuterVolumeSpecName: "kube-api-access-qlcpf") pod "e8d1fbfd-b455-4a35-80bb-398439b3e990" (UID: "e8d1fbfd-b455-4a35-80bb-398439b3e990"). InnerVolumeSpecName "kube-api-access-qlcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:27:11.054098 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.054058    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfpxq\" (UniqueName: \"kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq\") pod \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\" (UID: \"8b8968fd-46b0-4c0d-8a1c-e722970afea1\") "
Sep 09 12:27:11.054383 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.054363    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d1fbfd-b455-4a35-80bb-398439b3e990-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:11.054438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.054389    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlcpf\" (UniqueName: \"kubernetes.io/projected/e8d1fbfd-b455-4a35-80bb-398439b3e990-kube-api-access-qlcpf\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:11.056381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.056354    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq" (OuterVolumeSpecName: "kube-api-access-wfpxq") pod "8b8968fd-46b0-4c0d-8a1c-e722970afea1" (UID: "8b8968fd-46b0-4c0d-8a1c-e722970afea1"). InnerVolumeSpecName "kube-api-access-wfpxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:27:11.079519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.079485    2556 generic.go:356] "Generic (PLEG): container finished" podID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" containerID="e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6" exitCode=0
Sep 09 12:27:11.079708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.079536    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"
Sep 09 12:27:11.079708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.079573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" event={"ID":"8b8968fd-46b0-4c0d-8a1c-e722970afea1","Type":"ContainerDied","Data":"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"}
Sep 09 12:27:11.079708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.079640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj" event={"ID":"8b8968fd-46b0-4c0d-8a1c-e722970afea1","Type":"ContainerDied","Data":"b977d355f077e775d8e80a7087c19fff4584678a74ff6310f46af9fdc9d1f0a1"}
Sep 09 12:27:11.079708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.079676    2556 scope.go:117] "RemoveContainer" containerID="e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"
Sep 09 12:27:11.081191 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.081164    2556 generic.go:356] "Generic (PLEG): container finished" podID="e8d1fbfd-b455-4a35-80bb-398439b3e990" containerID="c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63" exitCode=0
Sep 09 12:27:11.081325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.081198    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"
Sep 09 12:27:11.081325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.081262    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" event={"ID":"e8d1fbfd-b455-4a35-80bb-398439b3e990","Type":"ContainerDied","Data":"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"}
Sep 09 12:27:11.081325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.081306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4" event={"ID":"e8d1fbfd-b455-4a35-80bb-398439b3e990","Type":"ContainerDied","Data":"25dfb876446906befd1d7567f3ab65a43d780848b0fe956629912f7a282d8ae0"}
Sep 09 12:27:11.090715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.090692    2556 scope.go:117] "RemoveContainer" containerID="e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"
Sep 09 12:27:11.091174 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:27:11.091034    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6\": container with ID starting with e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6 not found: ID does not exist" containerID="e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"
Sep 09 12:27:11.091174 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.091070    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6"} err="failed to get container status \"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6\": rpc error: code = NotFound desc = could not find container \"e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6\": container with ID starting with e9c76c0f187160db897989a8a091c0451f1c807e8ccc4365e71066cfa9164ef6 not found: ID does not exist"
Sep 09 12:27:11.091174 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.091095    2556 scope.go:117] "RemoveContainer" containerID="c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"
Sep 09 12:27:11.100313 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.100291    2556 scope.go:117] "RemoveContainer" containerID="c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"
Sep 09 12:27:11.100660 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:27:11.100640    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63\": container with ID starting with c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63 not found: ID does not exist" containerID="c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"
Sep 09 12:27:11.100739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.100669    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63"} err="failed to get container status \"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63\": rpc error: code = NotFound desc = could not find container \"c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63\": container with ID starting with c863d6fdd27145ef7db8b6a19d2f965decc8642f3ba4103ca65a1b383c404e63 not found: ID does not exist"
Sep 09 12:27:11.102885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.102860    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:27:11.109467 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.109443    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ebae-6d499bd64f-ktshj"]
Sep 09 12:27:11.122003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.121963    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:27:11.126644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.126614    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ebae-6b5546d6-q5qm4"]
Sep 09 12:27:11.155135 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.155039    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpxq\" (UniqueName: \"kubernetes.io/projected/8b8968fd-46b0-4c0d-8a1c-e722970afea1-kube-api-access-wfpxq\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:11.902185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.902145    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" path="/var/lib/kubelet/pods/8b8968fd-46b0-4c0d-8a1c-e722970afea1/volumes"
Sep 09 12:27:11.902605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:11.902587    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d1fbfd-b455-4a35-80bb-398439b3e990" path="/var/lib/kubelet/pods/e8d1fbfd-b455-4a35-80bb-398439b3e990/volumes"
Sep 09 12:27:17.897434 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:17.897369    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" podUID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" containerName="main" containerID="cri-o://6572b64038f5a8f8d374d4b046f23e065a48b4ad5c33c9f0269930e5819042b4" gracePeriod=45
Sep 09 12:27:18.119023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.118987    2556 generic.go:356] "Generic (PLEG): container finished" podID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" containerID="6572b64038f5a8f8d374d4b046f23e065a48b4ad5c33c9f0269930e5819042b4" exitCode=0
Sep 09 12:27:18.119212 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.119064    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" event={"ID":"cd69c41a-2af3-4665-b1a6-28f6a73119fc","Type":"ContainerDied","Data":"6572b64038f5a8f8d374d4b046f23e065a48b4ad5c33c9f0269930e5819042b4"}
Sep 09 12:27:18.158982 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.158957    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:27:18.216920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.216871    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.216938    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmgjm\" (UniqueName: \"kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.216981    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217014    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217275 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217210    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217345 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217312    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir\") pod \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\" (UID: \"cd69c41a-2af3-4665-b1a6-28f6a73119fc\") "
Sep 09 12:27:18.217399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217186    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:27:18.217399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217269    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home" (OuterVolumeSpecName: "home") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:27:18.217573 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:27:18.217632 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217620    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:18.217692 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.217639    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:18.219386 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.219366    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm" (OuterVolumeSpecName: "dshm") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:27:18.219451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.219432    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm" (OuterVolumeSpecName: "kube-api-access-gmgjm") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "kube-api-access-gmgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:27:18.219521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.219506    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd69c41a-2af3-4665-b1a6-28f6a73119fc" (UID: "cd69c41a-2af3-4665-b1a6-28f6a73119fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:27:18.319020 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.318969    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgjm\" (UniqueName: \"kubernetes.io/projected/cd69c41a-2af3-4665-b1a6-28f6a73119fc-kube-api-access-gmgjm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:18.319020 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.319012    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:18.319020 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.319024    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:18.319020 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:18.319033    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd69c41a-2af3-4665-b1a6-28f6a73119fc-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:27:19.125016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.124981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59" event={"ID":"cd69c41a-2af3-4665-b1a6-28f6a73119fc","Type":"ContainerDied","Data":"5da72ed23332872b86d7a724c9a10e5bb19b4eb1da2324156fffd15e8c5da638"}
Sep 09 12:27:19.125016 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.125016    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"
Sep 09 12:27:19.125469 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.125041    2556 scope.go:117] "RemoveContainer" containerID="6572b64038f5a8f8d374d4b046f23e065a48b4ad5c33c9f0269930e5819042b4"
Sep 09 12:27:19.146461 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.146438    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:27:19.152327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.152307    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a1e1f1/autores-model-a1e1f1-kserve-c78cd65dc-2zt59"]
Sep 09 12:27:19.901755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:19.901722    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" path="/var/lib/kubelet/pods/cd69c41a-2af3-4665-b1a6-28f6a73119fc/volumes"
Sep 09 12:27:27.340713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.340678    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341018    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" containerName="payload-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341029    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" containerName="payload-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341037    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d1fbfd-b455-4a35-80bb-398439b3e990" containerName="payload-pre-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341042    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d1fbfd-b455-4a35-80bb-398439b3e990" containerName="payload-pre-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341053    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c84a57-7501-42b1-adf0-15784d3a2b46" containerName="payload-pre-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341060    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c84a57-7501-42b1-adf0-15784d3a2b46" containerName="payload-pre-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341071    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" containerName="payload-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341076    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" containerName="payload-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341095    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" containerName="main"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341104    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" containerName="main"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341168    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8968fd-46b0-4c0d-8a1c-e722970afea1" containerName="payload-processing"
Sep 09 12:27:27.341185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341181    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c84a57-7501-42b1-adf0-15784d3a2b46" containerName="payload-pre-processing"
Sep 09 12:27:27.341606 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341192    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d1fbfd-b455-4a35-80bb-398439b3e990" containerName="payload-pre-processing"
Sep 09 12:27:27.341606 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341200    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea9f2a02-5cf7-4544-b4dd-c57a77bd1686" containerName="payload-processing"
Sep 09 12:27:27.341606 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.341206    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd69c41a-2af3-4665-b1a6-28f6a73119fc" containerName="main"
Sep 09 12:27:27.345849 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.345833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.348773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.348746    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b402ba\""
Sep 09 12:27:27.348910 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.348853    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b402ba-dockercfg-k4fvd\""
Sep 09 12:27:27.355746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.355717    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:27:27.358803 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.358782    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:27:27.362077 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.362056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.375722 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.375698    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:27:27.407183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.407150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx8c\" (UniqueName: \"kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.407373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.407217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4r5\" (UniqueName: \"kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.407373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.407319    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.407373 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.407339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.508521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.508480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4r5\" (UniqueName: \"kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.508751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.508572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.508751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.508598    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.508751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.508634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx8c\" (UniqueName: \"kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.509423 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.509395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.509521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.509447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.517707 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.517669    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4r5\" (UniqueName: \"kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5\") pod \"payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.517844 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.517763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx8c\" (UniqueName: \"kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c\") pod \"payload-processing-e2e-explicit-b402ba-669746769-ncr7s\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") " pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.657355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.657225    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:27.671766 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.671734    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:27.938427 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.938339    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:27:27.941860 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:27:27.941824    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136fca0d_af59_4e4c_96f4_74599f9d6397.slice/crio-bd54e350e7aaa60910669a843e342f9950736bd142a8e4f7846ca58af185712b WatchSource:0}: Error finding container bd54e350e7aaa60910669a843e342f9950736bd142a8e4f7846ca58af185712b: Status 404 returned error can't find the container with id bd54e350e7aaa60910669a843e342f9950736bd142a8e4f7846ca58af185712b
Sep 09 12:27:27.960483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:27.960459    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:27:27.965348 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:27:27.965320    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e41b890_cab5_46b9_9995_30f024663d87.slice/crio-e88bc9c31abf194ae9153cec69936a5b625b8a8417a245a87c5dfb0685e7c8c5 WatchSource:0}: Error finding container e88bc9c31abf194ae9153cec69936a5b625b8a8417a245a87c5dfb0685e7c8c5: Status 404 returned error can't find the container with id e88bc9c31abf194ae9153cec69936a5b625b8a8417a245a87c5dfb0685e7c8c5
Sep 09 12:27:28.170325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.170271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" event={"ID":"136fca0d-af59-4e4c-96f4-74599f9d6397","Type":"ContainerStarted","Data":"56d2b3ad97458b9c20a4c3174873bce154d9627958f01fd5e12d4c3d5e81bd37"}
Sep 09 12:27:28.170532 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.170343    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:27:28.170532 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.170367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" event={"ID":"136fca0d-af59-4e4c-96f4-74599f9d6397","Type":"ContainerStarted","Data":"bd54e350e7aaa60910669a843e342f9950736bd142a8e4f7846ca58af185712b"}
Sep 09 12:27:28.172009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.171980    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" event={"ID":"7e41b890-cab5-46b9-9995-30f024663d87","Type":"ContainerStarted","Data":"636a8debc6e787670def2ca7515641cbde0893005b356d7bff9540053c27cb22"}
Sep 09 12:27:28.172146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.172026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" event={"ID":"7e41b890-cab5-46b9-9995-30f024663d87","Type":"ContainerStarted","Data":"e88bc9c31abf194ae9153cec69936a5b625b8a8417a245a87c5dfb0685e7c8c5"}
Sep 09 12:27:28.172146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.172110    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:28.187717 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.187666    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" podStartSLOduration=1.187651478 podStartE2EDuration="1.187651478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:27:28.186188274 +0000 UTC m=+970.816776051" watchObservedRunningTime="2026-09-09 12:27:28.187651478 +0000 UTC m=+970.818239253"
Sep 09 12:27:28.205378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:28.205282    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" podStartSLOduration=1.205265842 podStartE2EDuration="1.205265842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:27:28.204432894 +0000 UTC m=+970.835020670" watchObservedRunningTime="2026-09-09 12:27:28.205265842 +0000 UTC m=+970.835853632"
Sep 09 12:27:39.179816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:39.179738    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:27:39.180187 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:27:39.179903    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:28:08.298891 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.298859    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:28:08.302728 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.302710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.305064 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.305042    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"premium-w4-bc0270-kserve-self-signed-certs\""
Sep 09 12:28:08.306036 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.306013    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"kube-root-ca.crt\""
Sep 09 12:28:08.306163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.306014    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"default-dockercfg-k469g\""
Sep 09 12:28:08.306163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.306015    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"openshift-service-ca.crt\""
Sep 09 12:28:08.314743 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.314719    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:28:08.381431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.381431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.381663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381477    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bswk\" (UniqueName: \"kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.381663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.381663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.381663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.381624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482591 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bswk\" (UniqueName: \"kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.482788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.482736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.483120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.483040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.483120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.483089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.483205 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.483140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.485164 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.485144    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.485458 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.485438    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.491043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.491019    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bswk\" (UniqueName: \"kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk\") pod \"premium-w4-bc0270-kserve-6c6b8dff58-hmss4\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") " pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.612784 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.612702    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:08.884564 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:08.884507    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:28:08.886765 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:28:08.886727    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86f893f2_bc6f_4afe_b71c_4348d9a0fa04.slice/crio-cf27dbbc7335c8ab30d2db48b65e8249b8d7439a302ce72099836f2505ec1408 WatchSource:0}: Error finding container cf27dbbc7335c8ab30d2db48b65e8249b8d7439a302ce72099836f2505ec1408: Status 404 returned error can't find the container with id cf27dbbc7335c8ab30d2db48b65e8249b8d7439a302ce72099836f2505ec1408
Sep 09 12:28:09.377816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:09.377781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" event={"ID":"86f893f2-bc6f-4afe-b71c-4348d9a0fa04","Type":"ContainerStarted","Data":"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"}
Sep 09 12:28:09.377816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:09.377820    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" event={"ID":"86f893f2-bc6f-4afe-b71c-4348d9a0fa04","Type":"ContainerStarted","Data":"cf27dbbc7335c8ab30d2db48b65e8249b8d7439a302ce72099836f2505ec1408"}
Sep 09 12:28:09.396796 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:09.396749    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" podStartSLOduration=1.396732952 podStartE2EDuration="1.396732952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:09.396218605 +0000 UTC m=+1012.026806372" watchObservedRunningTime="2026-09-09 12:28:09.396732952 +0000 UTC m=+1012.027320728"
Sep 09 12:28:18.612836 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:18.612792    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:18.613351 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:18.612884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:18.625665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:18.625633    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:19.434845 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:19.434812    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:28:36.139297 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.139266    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:28:36.143952 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.143934    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.146538 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.146509    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b402ba\"/\"kube-root-ca.crt\""
Sep 09 12:28:36.147489 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.147465    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b402ba\"/\"explicit-model-b402ba-kserve-self-signed-certs\""
Sep 09 12:28:36.147594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.147521    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b402ba\"/\"default-dockercfg-ld6qj\""
Sep 09 12:28:36.147594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.147516    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b402ba\"/\"openshift-service-ca.crt\""
Sep 09 12:28:36.162806 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.162770    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:28:36.241627 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.241793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.241793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241664    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.241793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241726    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.241793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.241936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.241797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2zn\" (UniqueName: \"kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342597    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2zn\" (UniqueName: \"kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.342976 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342949    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.343048 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.342991    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.343048 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.343041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.344990 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.344970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.345142 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.345127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.352363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.352334    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2zn\" (UniqueName: \"kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn\") pod \"explicit-model-b402ba-kserve-67c468f47f-dkhn4\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") " pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.453215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.453140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:36.728014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:36.727990    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:28:36.729622 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:28:36.729595    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf5361c2_4042_459b_9b0a_2c12ba7fadb7.slice/crio-0d6773a0dd59c64df7c67cc12e0cde581e524c848efdf69c973ccda2e186a79b WatchSource:0}: Error finding container 0d6773a0dd59c64df7c67cc12e0cde581e524c848efdf69c973ccda2e186a79b: Status 404 returned error can't find the container with id 0d6773a0dd59c64df7c67cc12e0cde581e524c848efdf69c973ccda2e186a79b
Sep 09 12:28:37.516520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:37.516485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" event={"ID":"bf5361c2-4042-459b-9b0a-2c12ba7fadb7","Type":"ContainerStarted","Data":"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"}
Sep 09 12:28:37.516520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:37.516526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" event={"ID":"bf5361c2-4042-459b-9b0a-2c12ba7fadb7","Type":"ContainerStarted","Data":"0d6773a0dd59c64df7c67cc12e0cde581e524c848efdf69c973ccda2e186a79b"}
Sep 09 12:28:37.536421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:37.536378    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" podStartSLOduration=1.536363647 podStartE2EDuration="1.536363647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:37.533914177 +0000 UTC m=+1040.164501953" watchObservedRunningTime="2026-09-09 12:28:37.536363647 +0000 UTC m=+1040.166951478"
Sep 09 12:28:46.453752 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:46.453715    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:46.454333 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:46.453987    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:46.466382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:46.466354    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:46.576078 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:46.576048    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:28:57.314684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.314651    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:28:57.318169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.318151    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.320592 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.320573    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a5f6d9c8\""
Sep 09 12:28:57.320746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.320730    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a5f6d9c8-dockercfg-p2758\""
Sep 09 12:28:57.326894 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.326871    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:28:57.352816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.352784    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:28:57.356121 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.356105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.365787 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.365765    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:28:57.419030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.418999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.419030 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.419034    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn69z\" (UniqueName: \"kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.419330 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.419110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jss6k\" (UniqueName: \"kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.419330 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.419173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.519988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.519943    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.519988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.519988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn69z\" (UniqueName: \"kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.520210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.520124    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jss6k\" (UniqueName: \"kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.520210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.520193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.520928 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.520902    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.521023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.521007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.529533 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.529510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jss6k\" (UniqueName: \"kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k\") pod \"payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.529742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.529724    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn69z\" (UniqueName: \"kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z\") pod \"payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") " pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.629513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.629422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:57.666665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.666633    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:57.907519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.907495    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:28:57.909971 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:28:57.909944    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd15953_c57c_45a8_8700_4b5af9658f26.slice/crio-428ab38b3f688d19f06bc89eac3ca0b63dd5e87daa3e76a531e3469ed29779b5 WatchSource:0}: Error finding container 428ab38b3f688d19f06bc89eac3ca0b63dd5e87daa3e76a531e3469ed29779b5: Status 404 returned error can't find the container with id 428ab38b3f688d19f06bc89eac3ca0b63dd5e87daa3e76a531e3469ed29779b5
Sep 09 12:28:57.959080 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:57.959053    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:28:57.960512 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:28:57.960484    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9fd74a_8475_4b7a_8d49_67fdc6fae461.slice/crio-c0f54a725027933e41a90583f9c49a286ad4366875671d4a8414b6506437f350 WatchSource:0}: Error finding container c0f54a725027933e41a90583f9c49a286ad4366875671d4a8414b6506437f350: Status 404 returned error can't find the container with id c0f54a725027933e41a90583f9c49a286ad4366875671d4a8414b6506437f350
Sep 09 12:28:58.625399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.625357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" event={"ID":"2d9fd74a-8475-4b7a-8d49-67fdc6fae461","Type":"ContainerStarted","Data":"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"}
Sep 09 12:28:58.625885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.625429    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:28:58.625885 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.625453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" event={"ID":"2d9fd74a-8475-4b7a-8d49-67fdc6fae461","Type":"ContainerStarted","Data":"c0f54a725027933e41a90583f9c49a286ad4366875671d4a8414b6506437f350"}
Sep 09 12:28:58.627083 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.627048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" event={"ID":"7dd15953-c57c-45a8-8700-4b5af9658f26","Type":"ContainerStarted","Data":"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"}
Sep 09 12:28:58.627209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.627097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" event={"ID":"7dd15953-c57c-45a8-8700-4b5af9658f26","Type":"ContainerStarted","Data":"428ab38b3f688d19f06bc89eac3ca0b63dd5e87daa3e76a531e3469ed29779b5"}
Sep 09 12:28:58.627209 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.627148    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:28:58.646163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.646114    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" podStartSLOduration=1.64609775 podStartE2EDuration="1.64609775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:58.644801925 +0000 UTC m=+1061.275389702" watchObservedRunningTime="2026-09-09 12:28:58.64609775 +0000 UTC m=+1061.276685526"
Sep 09 12:28:58.665264 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:28:58.665200    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" podStartSLOduration=1.665185079 podStartE2EDuration="1.665185079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:58.664215137 +0000 UTC m=+1061.294802912" watchObservedRunningTime="2026-09-09 12:28:58.665185079 +0000 UTC m=+1061.295772855"
Sep 09 12:29:05.689208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:05.689171    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:29:09.634575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:09.634482    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:29:09.634575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:09.634574    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:29:16.504134 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.504097    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:29:16.504572 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.504372    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" podUID="7e41b890-cab5-46b9-9995-30f024663d87" containerName="payload-processing" containerID="cri-o://636a8debc6e787670def2ca7515641cbde0893005b356d7bff9540053c27cb22" gracePeriod=30
Sep 09 12:29:16.509863 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.509836    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:29:16.510144 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.510079    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" podUID="136fca0d-af59-4e4c-96f4-74599f9d6397" containerName="payload-pre-processing" containerID="cri-o://56d2b3ad97458b9c20a4c3174873bce154d9627958f01fd5e12d4c3d5e81bd37" gracePeriod=30
Sep 09 12:29:16.721442 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.721400    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e41b890-cab5-46b9-9995-30f024663d87" containerID="636a8debc6e787670def2ca7515641cbde0893005b356d7bff9540053c27cb22" exitCode=0
Sep 09 12:29:16.721442 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.721436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" event={"ID":"7e41b890-cab5-46b9-9995-30f024663d87","Type":"ContainerDied","Data":"636a8debc6e787670def2ca7515641cbde0893005b356d7bff9540053c27cb22"}
Sep 09 12:29:16.723571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.723543    2556 generic.go:356] "Generic (PLEG): container finished" podID="136fca0d-af59-4e4c-96f4-74599f9d6397" containerID="56d2b3ad97458b9c20a4c3174873bce154d9627958f01fd5e12d4c3d5e81bd37" exitCode=0
Sep 09 12:29:16.723719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.723626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" event={"ID":"136fca0d-af59-4e4c-96f4-74599f9d6397","Type":"ContainerDied","Data":"56d2b3ad97458b9c20a4c3174873bce154d9627958f01fd5e12d4c3d5e81bd37"}
Sep 09 12:29:16.787720 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.787698    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:29:16.790521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.790504    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:29:16.897165 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897137    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hx8c\" (UniqueName: \"kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c\") pod \"7e41b890-cab5-46b9-9995-30f024663d87\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") "
Sep 09 12:29:16.897381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897207    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume\") pod \"7e41b890-cab5-46b9-9995-30f024663d87\" (UID: \"7e41b890-cab5-46b9-9995-30f024663d87\") "
Sep 09 12:29:16.897381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897293    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume\") pod \"136fca0d-af59-4e4c-96f4-74599f9d6397\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") "
Sep 09 12:29:16.897381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897340    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db4r5\" (UniqueName: \"kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5\") pod \"136fca0d-af59-4e4c-96f4-74599f9d6397\" (UID: \"136fca0d-af59-4e4c-96f4-74599f9d6397\") "
Sep 09 12:29:16.897642 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897606    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e41b890-cab5-46b9-9995-30f024663d87" (UID: "7e41b890-cab5-46b9-9995-30f024663d87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:29:16.897756 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.897637    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "136fca0d-af59-4e4c-96f4-74599f9d6397" (UID: "136fca0d-af59-4e4c-96f4-74599f9d6397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:29:16.899461 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.899435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c" (OuterVolumeSpecName: "kube-api-access-7hx8c") pod "7e41b890-cab5-46b9-9995-30f024663d87" (UID: "7e41b890-cab5-46b9-9995-30f024663d87"). InnerVolumeSpecName "kube-api-access-7hx8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:16.899546 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.899454    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5" (OuterVolumeSpecName: "kube-api-access-db4r5") pod "136fca0d-af59-4e4c-96f4-74599f9d6397" (UID: "136fca0d-af59-4e4c-96f4-74599f9d6397"). InnerVolumeSpecName "kube-api-access-db4r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:16.998664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.998633    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hx8c\" (UniqueName: \"kubernetes.io/projected/7e41b890-cab5-46b9-9995-30f024663d87-kube-api-access-7hx8c\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:16.998664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.998661    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e41b890-cab5-46b9-9995-30f024663d87-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:16.998839 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.998674    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136fca0d-af59-4e4c-96f4-74599f9d6397-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:16.998839 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:16.998686    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4r5\" (UniqueName: \"kubernetes.io/projected/136fca0d-af59-4e4c-96f4-74599f9d6397-kube-api-access-db4r5\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:17.730594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.730554    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"
Sep 09 12:29:17.731067 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.730562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s" event={"ID":"7e41b890-cab5-46b9-9995-30f024663d87","Type":"ContainerDied","Data":"e88bc9c31abf194ae9153cec69936a5b625b8a8417a245a87c5dfb0685e7c8c5"}
Sep 09 12:29:17.731067 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.730754    2556 scope.go:117] "RemoveContainer" containerID="636a8debc6e787670def2ca7515641cbde0893005b356d7bff9540053c27cb22"
Sep 09 12:29:17.732163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.732146    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"
Sep 09 12:29:17.732279 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.732150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2" event={"ID":"136fca0d-af59-4e4c-96f4-74599f9d6397","Type":"ContainerDied","Data":"bd54e350e7aaa60910669a843e342f9950736bd142a8e4f7846ca58af185712b"}
Sep 09 12:29:17.740706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.740688    2556 scope.go:117] "RemoveContainer" containerID="56d2b3ad97458b9c20a4c3174873bce154d9627958f01fd5e12d4c3d5e81bd37"
Sep 09 12:29:17.756112 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.756089    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:29:17.760668 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.760650    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b402ba-5d46d5df54-9qcc2"]
Sep 09 12:29:17.774572 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.774548    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:29:17.780336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.780314    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b402ba-669746769-ncr7s"]
Sep 09 12:29:17.902410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.902380    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136fca0d-af59-4e4c-96f4-74599f9d6397" path="/var/lib/kubelet/pods/136fca0d-af59-4e4c-96f4-74599f9d6397/volumes"
Sep 09 12:29:17.902834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:17.902819    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e41b890-cab5-46b9-9995-30f024663d87" path="/var/lib/kubelet/pods/7e41b890-cab5-46b9-9995-30f024663d87/volumes"
Sep 09 12:29:20.710299 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:20.710234    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" podUID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" containerName="main" containerID="cri-o://81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2" gracePeriod=45
Sep 09 12:29:20.966450 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:20.966390    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:29:21.037584 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037553    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.037772 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.037772 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037660    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.037772 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037725    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.037948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037778    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t2zn\" (UniqueName: \"kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.037948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037805    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache\") pod \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\" (UID: \"bf5361c2-4042-459b-9b0a-2c12ba7fadb7\") "
Sep 09 12:29:21.038050 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.037957    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home" (OuterVolumeSpecName: "home") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:29:21.038050 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.038003    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:29:21.038143 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.038112    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:29:21.038447 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.038326    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.038447 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.038355    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.038447 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.038368    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.040172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.040149    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:29:21.040305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.040183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn" (OuterVolumeSpecName: "kube-api-access-4t2zn") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "kube-api-access-4t2zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:21.040305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.040211    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm" (OuterVolumeSpecName: "dshm") pod "bf5361c2-4042-459b-9b0a-2c12ba7fadb7" (UID: "bf5361c2-4042-459b-9b0a-2c12ba7fadb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:29:21.139090 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.139058    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.139090 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.139088    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2zn\" (UniqueName: \"kubernetes.io/projected/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-kube-api-access-4t2zn\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.139328 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.139100    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5361c2-4042-459b-9b0a-2c12ba7fadb7-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:21.754808 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.754771    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" containerID="81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2" exitCode=0
Sep 09 12:29:21.755327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.754817    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"
Sep 09 12:29:21.755327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.754855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" event={"ID":"bf5361c2-4042-459b-9b0a-2c12ba7fadb7","Type":"ContainerDied","Data":"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"}
Sep 09 12:29:21.755327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.754897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4" event={"ID":"bf5361c2-4042-459b-9b0a-2c12ba7fadb7","Type":"ContainerDied","Data":"0d6773a0dd59c64df7c67cc12e0cde581e524c848efdf69c973ccda2e186a79b"}
Sep 09 12:29:21.755327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.754912    2556 scope.go:117] "RemoveContainer" containerID="81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"
Sep 09 12:29:21.764108 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.764089    2556 scope.go:117] "RemoveContainer" containerID="81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"
Sep 09 12:29:21.764441 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:29:21.764421    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2\": container with ID starting with 81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2 not found: ID does not exist" containerID="81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"
Sep 09 12:29:21.764517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.764451    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2"} err="failed to get container status \"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2\": rpc error: code = NotFound desc = could not find container \"81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2\": container with ID starting with 81ca6be8e27a8b9e8d137de5aed070eaaf6c0aaca8227e5e52cd89bde7fc07b2 not found: ID does not exist"
Sep 09 12:29:21.780620 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.780584    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:29:21.789279 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.789252    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b402ba/explicit-model-b402ba-kserve-67c468f47f-dkhn4"]
Sep 09 12:29:21.903881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.903849    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" path="/var/lib/kubelet/pods/bf5361c2-4042-459b-9b0a-2c12ba7fadb7/volumes"
Sep 09 12:29:21.909205 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909178    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:29:21.909601 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909584    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136fca0d-af59-4e4c-96f4-74599f9d6397" containerName="payload-pre-processing"
Sep 09 12:29:21.909683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909604    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="136fca0d-af59-4e4c-96f4-74599f9d6397" containerName="payload-pre-processing"
Sep 09 12:29:21.909683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909620    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e41b890-cab5-46b9-9995-30f024663d87" containerName="payload-processing"
Sep 09 12:29:21.909683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909629    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e41b890-cab5-46b9-9995-30f024663d87" containerName="payload-processing"
Sep 09 12:29:21.909683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909641    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" containerName="main"
Sep 09 12:29:21.909683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909650    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" containerName="main"
Sep 09 12:29:21.909925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909756    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e41b890-cab5-46b9-9995-30f024663d87" containerName="payload-processing"
Sep 09 12:29:21.909925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909770    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5361c2-4042-459b-9b0a-2c12ba7fadb7" containerName="main"
Sep 09 12:29:21.909925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.909779    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="136fca0d-af59-4e4c-96f4-74599f9d6397" containerName="payload-pre-processing"
Sep 09 12:29:21.914050 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.914031    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:21.916509 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.916490    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"distinct-w4-bc0270-kserve-self-signed-certs\""
Sep 09 12:29:21.921855 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:21.921835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:29:22.047563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047471    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.047747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.047747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qcx\" (UniqueName: \"kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.047747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.047747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.047955 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.047758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149283 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qcx\" (UniqueName: \"kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149796 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149773    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.149944 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.150027 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.149944    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.151751 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.151720    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.151942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.151926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.157574 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.157554    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qcx\" (UniqueName: \"kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx\") pod \"distinct-w4-bc0270-kserve-544b6db76f-gfknn\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") " pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.225931 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.225884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:22.497899 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.497873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:29:22.499600 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:29:22.499572    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71e2bc3_ef89_4f0d_8921_56f6ad445bcb.slice/crio-ce0023187dd59445f81bf8e032948ae02bc4956564e91d508b95fefb8877a590 WatchSource:0}: Error finding container ce0023187dd59445f81bf8e032948ae02bc4956564e91d508b95fefb8877a590: Status 404 returned error can't find the container with id ce0023187dd59445f81bf8e032948ae02bc4956564e91d508b95fefb8877a590
Sep 09 12:29:22.761344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.761229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" event={"ID":"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb","Type":"ContainerStarted","Data":"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"}
Sep 09 12:29:22.761344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.761301    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" event={"ID":"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb","Type":"ContainerStarted","Data":"ce0023187dd59445f81bf8e032948ae02bc4956564e91d508b95fefb8877a590"}
Sep 09 12:29:22.779302 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:22.779252    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" podStartSLOduration=1.7792231429999998 podStartE2EDuration="1.779223143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:29:22.777266901 +0000 UTC m=+1085.407854675" watchObservedRunningTime="2026-09-09 12:29:22.779223143 +0000 UTC m=+1085.409810919"
Sep 09 12:29:28.591149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.591106    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"]
Sep 09 12:29:28.594895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.594873    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:28.597275 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.597252    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:29:28.597398 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.597231    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:29:28.603310 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.603284    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"]
Sep 09 12:29:28.709895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.709853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz\" (UID: \"54f0c3f0-1e50-4542-8857-c07124ddc0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:28.811065 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.811033    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz\" (UID: \"54f0c3f0-1e50-4542-8857-c07124ddc0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:28.811664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.811643    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz\" (UID: \"54f0c3f0-1e50-4542-8857-c07124ddc0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:28.907559 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:28.907485    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:29.177976 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:29.177944    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"]
Sep 09 12:29:29.180114 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:29:29.180078    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f0c3f0_1e50_4542_8857_c07124ddc0f3.slice/crio-529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96 WatchSource:0}: Error finding container 529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96: Status 404 returned error can't find the container with id 529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96
Sep 09 12:29:29.182274 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:29.182256    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:29:29.795353 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:29.795315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz" event={"ID":"54f0c3f0-1e50-4542-8857-c07124ddc0f3","Type":"ContainerStarted","Data":"529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96"}
Sep 09 12:29:32.226888 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.226856    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:32.227362 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.226914    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:32.239585 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.239559    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:32.812792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.812756    2556 generic.go:356] "Generic (PLEG): container finished" podID="54f0c3f0-1e50-4542-8857-c07124ddc0f3" containerID="5135db83c6ae93985723f7376a8f6ce64aa188a768e7953fee552d595ca91b3b" exitCode=0
Sep 09 12:29:32.812965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.812848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz" event={"ID":"54f0c3f0-1e50-4542-8857-c07124ddc0f3","Type":"ContainerDied","Data":"5135db83c6ae93985723f7376a8f6ce64aa188a768e7953fee552d595ca91b3b"}
Sep 09 12:29:32.824216 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:32.824184    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:29:33.950342 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:33.950317    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:34.067825 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.067788    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca\") pod \"54f0c3f0-1e50-4542-8857-c07124ddc0f3\" (UID: \"54f0c3f0-1e50-4542-8857-c07124ddc0f3\") "
Sep 09 12:29:34.068143 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.068120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54f0c3f0-1e50-4542-8857-c07124ddc0f3" (UID: "54f0c3f0-1e50-4542-8857-c07124ddc0f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:29:34.168944 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.168863    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f0c3f0-1e50-4542-8857-c07124ddc0f3-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:34.824217 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.824189    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"
Sep 09 12:29:34.824421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.824223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz" event={"ID":"54f0c3f0-1e50-4542-8857-c07124ddc0f3","Type":"ContainerDied","Data":"529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96"}
Sep 09 12:29:34.824421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:34.824287    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="529a88e9736251b2513085d2c87b70eea8ce27b9bb291e160ac39673957c5b96"
Sep 09 12:29:39.217424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.217392    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:29:39.217947 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.217610    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" podUID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" containerName="payload-processing" containerID="cri-o://e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8" gracePeriod=30
Sep 09 12:29:39.223957 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.223913    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:29:39.224215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.224136    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" podUID="7dd15953-c57c-45a8-8700-4b5af9658f26" containerName="payload-pre-processing" containerID="cri-o://052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49" gracePeriod=30
Sep 09 12:29:39.400815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.400778    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"]
Sep 09 12:29:39.404346 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.404322    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5f6d9c8-k9jjz"]
Sep 09 12:29:39.494080 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.494048    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:29:39.496859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.496834    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:29:39.520959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.520936    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn69z\" (UniqueName: \"kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z\") pod \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") "
Sep 09 12:29:39.521109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.520977    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume\") pod \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\" (UID: \"2d9fd74a-8475-4b7a-8d49-67fdc6fae461\") "
Sep 09 12:29:39.521109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.521100    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume\") pod \"7dd15953-c57c-45a8-8700-4b5af9658f26\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") "
Sep 09 12:29:39.521234 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.521128    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jss6k\" (UniqueName: \"kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k\") pod \"7dd15953-c57c-45a8-8700-4b5af9658f26\" (UID: \"7dd15953-c57c-45a8-8700-4b5af9658f26\") "
Sep 09 12:29:39.521396 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.521370    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d9fd74a-8475-4b7a-8d49-67fdc6fae461" (UID: "2d9fd74a-8475-4b7a-8d49-67fdc6fae461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:29:39.521507 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.521482    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:39.521550 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.521514    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7dd15953-c57c-45a8-8700-4b5af9658f26" (UID: "7dd15953-c57c-45a8-8700-4b5af9658f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:29:39.523344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.523310    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k" (OuterVolumeSpecName: "kube-api-access-jss6k") pod "7dd15953-c57c-45a8-8700-4b5af9658f26" (UID: "7dd15953-c57c-45a8-8700-4b5af9658f26"). InnerVolumeSpecName "kube-api-access-jss6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:39.523435 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.523346    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z" (OuterVolumeSpecName: "kube-api-access-nn69z") pod "2d9fd74a-8475-4b7a-8d49-67fdc6fae461" (UID: "2d9fd74a-8475-4b7a-8d49-67fdc6fae461"). InnerVolumeSpecName "kube-api-access-nn69z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:29:39.621925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.621894    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn69z\" (UniqueName: \"kubernetes.io/projected/2d9fd74a-8475-4b7a-8d49-67fdc6fae461-kube-api-access-nn69z\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:39.621925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.621920    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd15953-c57c-45a8-8700-4b5af9658f26-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:39.621925 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.621931    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jss6k\" (UniqueName: \"kubernetes.io/projected/7dd15953-c57c-45a8-8700-4b5af9658f26-kube-api-access-jss6k\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:29:39.849610 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.849514    2556 generic.go:356] "Generic (PLEG): container finished" podID="7dd15953-c57c-45a8-8700-4b5af9658f26" containerID="052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49" exitCode=0
Sep 09 12:29:39.849610 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.849562    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"
Sep 09 12:29:39.849855 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.849603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" event={"ID":"7dd15953-c57c-45a8-8700-4b5af9658f26","Type":"ContainerDied","Data":"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"}
Sep 09 12:29:39.849855 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.849664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct" event={"ID":"7dd15953-c57c-45a8-8700-4b5af9658f26","Type":"ContainerDied","Data":"428ab38b3f688d19f06bc89eac3ca0b63dd5e87daa3e76a531e3469ed29779b5"}
Sep 09 12:29:39.849855 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.849693    2556 scope.go:117] "RemoveContainer" containerID="052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"
Sep 09 12:29:39.851025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.851005    2556 generic.go:356] "Generic (PLEG): container finished" podID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" containerID="e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8" exitCode=0
Sep 09 12:29:39.851128 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.851051    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"
Sep 09 12:29:39.851128 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.851083    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" event={"ID":"2d9fd74a-8475-4b7a-8d49-67fdc6fae461","Type":"ContainerDied","Data":"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"}
Sep 09 12:29:39.851225 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.851130    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8" event={"ID":"2d9fd74a-8475-4b7a-8d49-67fdc6fae461","Type":"ContainerDied","Data":"c0f54a725027933e41a90583f9c49a286ad4366875671d4a8414b6506437f350"}
Sep 09 12:29:39.858785 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.858680    2556 scope.go:117] "RemoveContainer" containerID="052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"
Sep 09 12:29:39.859092 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:29:39.859072    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49\": container with ID starting with 052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49 not found: ID does not exist" containerID="052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"
Sep 09 12:29:39.859194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.859101    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49"} err="failed to get container status \"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49\": rpc error: code = NotFound desc = could not find container \"052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49\": container with ID starting with 052260ff1fcb2409b03333f3c0dc920f31993241c6ec80c333fad6203fefee49 not found: ID does not exist"
Sep 09 12:29:39.859194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.859124    2556 scope.go:117] "RemoveContainer" containerID="e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"
Sep 09 12:29:39.867701 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.867678    2556 scope.go:117] "RemoveContainer" containerID="e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"
Sep 09 12:29:39.867976 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:29:39.867957    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8\": container with ID starting with e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8 not found: ID does not exist" containerID="e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"
Sep 09 12:29:39.868040 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.867980    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8"} err="failed to get container status \"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8\": rpc error: code = NotFound desc = could not find container \"e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8\": container with ID starting with e61b38e41c6eeb7bba4d06586cdddd9086febc3489672ee4fe62636d579f6ab8 not found: ID does not exist"
Sep 09 12:29:39.875094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.875074    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:29:39.879334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.879314    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a5f6d9c8-6d7f75ff54-5kbct"]
Sep 09 12:29:39.890246 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.890218    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:29:39.892406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.892386    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a5f6d9c8-6c5b85d969-9vbn8"]
Sep 09 12:29:39.901559 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.901535    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" path="/var/lib/kubelet/pods/2d9fd74a-8475-4b7a-8d49-67fdc6fae461/volumes"
Sep 09 12:29:39.901930 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.901912    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f0c3f0-1e50-4542-8857-c07124ddc0f3" path="/var/lib/kubelet/pods/54f0c3f0-1e50-4542-8857-c07124ddc0f3/volumes"
Sep 09 12:29:39.902325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:29:39.902308    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd15953-c57c-45a8-8700-4b5af9658f26" path="/var/lib/kubelet/pods/7dd15953-c57c-45a8-8700-4b5af9658f26/volumes"
Sep 09 12:30:00.154622 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154585    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"]
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154945    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f0c3f0-1e50-4542-8857-c07124ddc0f3" containerName="revoke-keys"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154957    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f0c3f0-1e50-4542-8857-c07124ddc0f3" containerName="revoke-keys"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154971    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd15953-c57c-45a8-8700-4b5af9658f26" containerName="payload-pre-processing"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154976    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd15953-c57c-45a8-8700-4b5af9658f26" containerName="payload-pre-processing"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.154995    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" containerName="payload-processing"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.155000    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" containerName="payload-processing"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.155053    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9fd74a-8475-4b7a-8d49-67fdc6fae461" containerName="payload-processing"
Sep 09 12:30:00.155062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.155064    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f0c3f0-1e50-4542-8857-c07124ddc0f3" containerName="revoke-keys"
Sep 09 12:30:00.155341 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.155072    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd15953-c57c-45a8-8700-4b5af9658f26" containerName="payload-pre-processing"
Sep 09 12:30:00.157956 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.157936    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"
Sep 09 12:30:00.160386 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.160370    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:30:00.168256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.168217    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"]
Sep 09 12:30:00.168391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.168367    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"
Sep 09 12:30:00.172214 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.172194    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"]
Sep 09 12:30:00.172365 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.172345    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"
Sep 09 12:30:00.175963 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.175941    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"]
Sep 09 12:30:00.176058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.176032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"
Sep 09 12:30:00.183169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.182766    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"]
Sep 09 12:30:00.185803 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.185587    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"]
Sep 09 12:30:00.206231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.206191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"
Sep 09 12:30:00.213347 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.213324    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"
Sep 09 12:30:00.456100 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.456074    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"]
Sep 09 12:30:00.458280 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:00.458227    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode971febc_8717_446d_a736_7b19271326c1.slice/crio-118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0 WatchSource:0}: Error finding container 118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0: Status 404 returned error can't find the container with id 118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0
Sep 09 12:30:00.493911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.493888    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"]
Sep 09 12:30:00.495734 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:00.495706    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a53b9de_7f7c_49eb_aba9_85fce2d4b9ed.slice/crio-518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a WatchSource:0}: Error finding container 518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a: Status 404 returned error can't find the container with id 518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a
Sep 09 12:30:00.519659 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.519639    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"]
Sep 09 12:30:00.530252 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:00.530215    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6557f0_0ca8_4766_afff_2f555904a3aa.slice/crio-ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d WatchSource:0}: Error finding container ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d: Status 404 returned error can't find the container with id ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d
Sep 09 12:30:00.968512 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.968478    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs" event={"ID":"0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed","Type":"ContainerStarted","Data":"77a4e4da824a5926c49fac37c16c99889df194cb93ae441be9ab6c8ce39e8787"}
Sep 09 12:30:00.968713 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.968536    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs" event={"ID":"0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed","Type":"ContainerStarted","Data":"518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a"}
Sep 09 12:30:00.970207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.970180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw" event={"ID":"9f6557f0-0ca8-4766-afff-2f555904a3aa","Type":"ContainerStarted","Data":"aa413827d274d75305d95cad479bdcc9797107d600d707d5dc2adbf56daf740b"}
Sep 09 12:30:00.970360 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.970233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw" event={"ID":"9f6557f0-0ca8-4766-afff-2f555904a3aa","Type":"ContainerStarted","Data":"ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d"}
Sep 09 12:30:00.971669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.971641    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt" event={"ID":"e971febc-8717-446d-a736-7b19271326c1","Type":"ContainerStarted","Data":"c9aed67b54b4a8e97e31c7c5d63d9c71373266ae837fcd54b646521f4a6004b1"}
Sep 09 12:30:00.971790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.971707    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt" event={"ID":"e971febc-8717-446d-a736-7b19271326c1","Type":"ContainerStarted","Data":"118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0"}
Sep 09 12:30:00.984201 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.984163    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs" podStartSLOduration=0.984148104 podStartE2EDuration="984.148104ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:00.983534409 +0000 UTC m=+1123.614122185" watchObservedRunningTime="2026-09-09 12:30:00.984148104 +0000 UTC m=+1123.614735878"
Sep 09 12:30:00.998649 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:00.998607    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw" podStartSLOduration=0.998592794 podStartE2EDuration="998.592794ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:00.997689539 +0000 UTC m=+1123.628277315" watchObservedRunningTime="2026-09-09 12:30:00.998592794 +0000 UTC m=+1123.629180570"
Sep 09 12:30:01.013732 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.013688    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt" podStartSLOduration=1.013674469 podStartE2EDuration="1.013674469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:01.011963903 +0000 UTC m=+1123.642551679" watchObservedRunningTime="2026-09-09 12:30:01.013674469 +0000 UTC m=+1123.644262243"
Sep 09 12:30:01.977900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.977869    2556 generic.go:356] "Generic (PLEG): container finished" podID="0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed" containerID="77a4e4da824a5926c49fac37c16c99889df194cb93ae441be9ab6c8ce39e8787" exitCode=0
Sep 09 12:30:01.978321 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.977957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs" event={"ID":"0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed","Type":"ContainerDied","Data":"77a4e4da824a5926c49fac37c16c99889df194cb93ae441be9ab6c8ce39e8787"}
Sep 09 12:30:01.979533 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.979510    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f6557f0-0ca8-4766-afff-2f555904a3aa" containerID="aa413827d274d75305d95cad479bdcc9797107d600d707d5dc2adbf56daf740b" exitCode=0
Sep 09 12:30:01.979652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.979584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw" event={"ID":"9f6557f0-0ca8-4766-afff-2f555904a3aa","Type":"ContainerDied","Data":"aa413827d274d75305d95cad479bdcc9797107d600d707d5dc2adbf56daf740b"}
Sep 09 12:30:01.980959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.980939    2556 generic.go:356] "Generic (PLEG): container finished" podID="e971febc-8717-446d-a736-7b19271326c1" containerID="c9aed67b54b4a8e97e31c7c5d63d9c71373266ae837fcd54b646521f4a6004b1" exitCode=0
Sep 09 12:30:01.981049 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:01.980994    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt" event={"ID":"e971febc-8717-446d-a736-7b19271326c1","Type":"ContainerDied","Data":"c9aed67b54b4a8e97e31c7c5d63d9c71373266ae837fcd54b646521f4a6004b1"}
Sep 09 12:30:03.137385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.137355    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"
Sep 09 12:30:03.179369 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.179344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"
Sep 09 12:30:03.182107 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.182090    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"
Sep 09 12:30:03.992250 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.992218    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"
Sep 09 12:30:03.992398 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.992229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs" event={"ID":"0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed","Type":"ContainerDied","Data":"518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a"}
Sep 09 12:30:03.992398 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.992299    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518ce3fe51afa95331a77c451ed9038640337d0b0dd851cb296c54955a0da27a"
Sep 09 12:30:03.993553 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.993522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw" event={"ID":"9f6557f0-0ca8-4766-afff-2f555904a3aa","Type":"ContainerDied","Data":"ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d"}
Sep 09 12:30:03.993701 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.993569    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef63ac5ebf1db6d571d9c82d196354b43dcb070586e72d5ef0cc2d8227b9ea4d"
Sep 09 12:30:03.993701 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.993532    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"
Sep 09 12:30:03.995158 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.995128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt" event={"ID":"e971febc-8717-446d-a736-7b19271326c1","Type":"ContainerDied","Data":"118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0"}
Sep 09 12:30:03.995158 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.995156    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="118208ae1db2a563f3bfc32ef66ffa7c62081c3a4b31ced8b17226b206b726b0"
Sep 09 12:30:03.995158 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:03.995135    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"
Sep 09 12:30:06.480981 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.480946    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:30:06.481391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481350    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6557f0-0ca8-4766-afff-2f555904a3aa" containerName="cleanup"
Sep 09 12:30:06.481391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481362    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6557f0-0ca8-4766-afff-2f555904a3aa" containerName="cleanup"
Sep 09 12:30:06.481391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481377    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e971febc-8717-446d-a736-7b19271326c1" containerName="cleanup"
Sep 09 12:30:06.481391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481384    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e971febc-8717-446d-a736-7b19271326c1" containerName="cleanup"
Sep 09 12:30:06.481391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481391    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed" containerName="cleanup"
Sep 09 12:30:06.481552 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481399    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed" containerName="cleanup"
Sep 09 12:30:06.481552 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481467    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed" containerName="cleanup"
Sep 09 12:30:06.481552 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481476    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6557f0-0ca8-4766-afff-2f555904a3aa" containerName="cleanup"
Sep 09 12:30:06.481552 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.481481    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e971febc-8717-446d-a736-7b19271326c1" containerName="cleanup"
Sep 09 12:30:06.484802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.484786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.487109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.487086    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d1f639-dockercfg-hcw5d\""
Sep 09 12:30:06.487109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.487100    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d1f639\""
Sep 09 12:30:06.493287 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.493264    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:30:06.529588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.529558    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:30:06.533009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.532994    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.545009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.544983    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:30:06.558086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.558061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.558217 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.558095    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.659350 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.659307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.659519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.659357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2tt\" (UniqueName: \"kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.659519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.659391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.659519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.659456    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.660125 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.660102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.668795 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.668775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v\") pod \"payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.760086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.760006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.760086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.760046    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2tt\" (UniqueName: \"kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.760735 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.760715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.769651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.769618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2tt\" (UniqueName: \"kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt\") pod \"payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:06.795960 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.795936    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:06.842782 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:06.842737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:07.071086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:07.071058    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:30:07.072479 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:07.072457    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad98c68_276b_4aee_8282_dbb70fca298c.slice/crio-acf606943a0dc72f6e71d7fae8b6a68687b8a4c1b902a19666cc30048aa198dd WatchSource:0}: Error finding container acf606943a0dc72f6e71d7fae8b6a68687b8a4c1b902a19666cc30048aa198dd: Status 404 returned error can't find the container with id acf606943a0dc72f6e71d7fae8b6a68687b8a4c1b902a19666cc30048aa198dd
Sep 09 12:30:07.132558 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:07.132532    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:30:07.134074 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:07.134048    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d40b59_5e24_46ef_85dd_7f85bd902430.slice/crio-c4ae0cbb696e8d64c5fc437c428d132394829ac03c5584069ec0beebe42da93b WatchSource:0}: Error finding container c4ae0cbb696e8d64c5fc437c428d132394829ac03c5584069ec0beebe42da93b: Status 404 returned error can't find the container with id c4ae0cbb696e8d64c5fc437c428d132394829ac03c5584069ec0beebe42da93b
Sep 09 12:30:08.021999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.021954    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" event={"ID":"6ad98c68-276b-4aee-8282-dbb70fca298c","Type":"ContainerStarted","Data":"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"}
Sep 09 12:30:08.022487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.022019    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" event={"ID":"6ad98c68-276b-4aee-8282-dbb70fca298c","Type":"ContainerStarted","Data":"acf606943a0dc72f6e71d7fae8b6a68687b8a4c1b902a19666cc30048aa198dd"}
Sep 09 12:30:08.022487 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.022045    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:08.023585 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.023554    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" event={"ID":"45d40b59-5e24-46ef-85dd-7f85bd902430","Type":"ContainerStarted","Data":"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"}
Sep 09 12:30:08.023720 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.023598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" event={"ID":"45d40b59-5e24-46ef-85dd-7f85bd902430","Type":"ContainerStarted","Data":"c4ae0cbb696e8d64c5fc437c428d132394829ac03c5584069ec0beebe42da93b"}
Sep 09 12:30:08.023720 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.023644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:08.043815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.043766    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" podStartSLOduration=2.043753055 podStartE2EDuration="2.043753055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:08.04234468 +0000 UTC m=+1130.672932456" watchObservedRunningTime="2026-09-09 12:30:08.043753055 +0000 UTC m=+1130.674340831"
Sep 09 12:30:08.059658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:08.059611    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" podStartSLOduration=2.059596626 podStartE2EDuration="2.059596626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:08.057727493 +0000 UTC m=+1130.688315269" watchObservedRunningTime="2026-09-09 12:30:08.059596626 +0000 UTC m=+1130.690184400"
Sep 09 12:30:19.030469 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:19.030438    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:30:19.030901 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:19.030707    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:30:35.836643 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.836607    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:30:35.841502 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.841483    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:35.843892 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.843870    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4ec60c\""
Sep 09 12:30:35.850665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.850635    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:30:35.872160 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.872129    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:30:35.875500 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.875484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:35.878099 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.878078    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4ec60c-dockercfg-d6nft\""
Sep 09 12:30:35.886346 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.886320    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:30:35.939595 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.939558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:35.939792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.939624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcw8\" (UniqueName: \"kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:35.939792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.939652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq756\" (UniqueName: \"kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:35.939792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:35.939742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.040662 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.040625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:36.040859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.040683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcw8\" (UniqueName: \"kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.040859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.040807    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq756\" (UniqueName: \"kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:36.040859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.040855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.041561 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.041542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.041651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.041596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:36.052869 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.052839    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcw8\" (UniqueName: \"kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8\") pod \"payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.052869 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.052850    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq756\" (UniqueName: \"kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756\") pod \"payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:36.153109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.153035    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:36.185594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.185563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:36.431039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.431011    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:30:36.432856 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:36.432835    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb55623_a220_41a1_804e_d88007016ee8.slice/crio-e195e952a276f8d4aeac7613292338f26c826383a66db0be361d3f7c79a7d401 WatchSource:0}: Error finding container e195e952a276f8d4aeac7613292338f26c826383a66db0be361d3f7c79a7d401: Status 404 returned error can't find the container with id e195e952a276f8d4aeac7613292338f26c826383a66db0be361d3f7c79a7d401
Sep 09 12:30:36.473388 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:36.473362    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:30:36.475069 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:36.475037    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555452be_2547_4ae6_b0ac_ff3dc4312cf0.slice/crio-69ab5d7821e079b46a3e9f4534bc086405b2fa565f84e27111d19a341b4c1c9a WatchSource:0}: Error finding container 69ab5d7821e079b46a3e9f4534bc086405b2fa565f84e27111d19a341b4c1c9a: Status 404 returned error can't find the container with id 69ab5d7821e079b46a3e9f4534bc086405b2fa565f84e27111d19a341b4c1c9a
Sep 09 12:30:37.167383 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.167344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" event={"ID":"ffb55623-a220-41a1-804e-d88007016ee8","Type":"ContainerStarted","Data":"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"}
Sep 09 12:30:37.167873 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.167440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" event={"ID":"ffb55623-a220-41a1-804e-d88007016ee8","Type":"ContainerStarted","Data":"e195e952a276f8d4aeac7613292338f26c826383a66db0be361d3f7c79a7d401"}
Sep 09 12:30:37.167873 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.167605    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:37.169301 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.169278    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" event={"ID":"555452be-2547-4ae6-b0ac-ff3dc4312cf0","Type":"ContainerStarted","Data":"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"}
Sep 09 12:30:37.169429 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.169316    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" event={"ID":"555452be-2547-4ae6-b0ac-ff3dc4312cf0","Type":"ContainerStarted","Data":"69ab5d7821e079b46a3e9f4534bc086405b2fa565f84e27111d19a341b4c1c9a"}
Sep 09 12:30:37.169429 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.169352    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:37.187504 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.187448    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" podStartSLOduration=2.187432887 podStartE2EDuration="2.187432887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:37.184975244 +0000 UTC m=+1159.815563019" watchObservedRunningTime="2026-09-09 12:30:37.187432887 +0000 UTC m=+1159.818020662"
Sep 09 12:30:37.209916 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:37.209861    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" podStartSLOduration=2.209846064 podStartE2EDuration="2.209846064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:37.207582999 +0000 UTC m=+1159.838170793" watchObservedRunningTime="2026-09-09 12:30:37.209846064 +0000 UTC m=+1159.840433840"
Sep 09 12:30:41.196039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.195956    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:30:41.199730 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.199713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.202247 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.202214    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-db67bc\"/\"kube-root-ca.crt\""
Sep 09 12:30:41.202346 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.202327    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-db67bc\"/\"openshift-service-ca.crt\""
Sep 09 12:30:41.203004 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.202988    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-db67bc\"/\"default-dockercfg-4bwbz\""
Sep 09 12:30:41.203277 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.203263    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-db67bc\"/\"noten-model-db67bc-kserve-self-signed-certs\""
Sep 09 12:30:41.210098 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.210071    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:30:41.289542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289507    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.289715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79q7\" (UniqueName: \"kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.289715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.289715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.289715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.289859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.289718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391106 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391118    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79q7\" (UniqueName: \"kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391325 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391559 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391726 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.391726 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391719    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.392075 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.391734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.393661 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.393635    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.393840 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.393822    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.405140 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.405118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79q7\" (UniqueName: \"kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7\") pod \"noten-model-db67bc-kserve-7cb65f4795-c5rpw\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") " pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.511149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.511108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:41.791818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:41.791712    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:30:41.794419 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:30:41.794376    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod927f1a60_16dd_4864_a836_b40eb0eb7aa8.slice/crio-19896de72f78c2fc0c7d22226b8bacdd792ab5617889db40d6b4c5fef9ad6f9b WatchSource:0}: Error finding container 19896de72f78c2fc0c7d22226b8bacdd792ab5617889db40d6b4c5fef9ad6f9b: Status 404 returned error can't find the container with id 19896de72f78c2fc0c7d22226b8bacdd792ab5617889db40d6b4c5fef9ad6f9b
Sep 09 12:30:42.196116 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:42.196022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" event={"ID":"927f1a60-16dd-4864-a836-b40eb0eb7aa8","Type":"ContainerStarted","Data":"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"}
Sep 09 12:30:42.196116 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:42.196070    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" event={"ID":"927f1a60-16dd-4864-a836-b40eb0eb7aa8","Type":"ContainerStarted","Data":"19896de72f78c2fc0c7d22226b8bacdd792ab5617889db40d6b4c5fef9ad6f9b"}
Sep 09 12:30:42.225292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:42.225223    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" podStartSLOduration=1.225206678 podStartE2EDuration="1.225206678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:30:42.222838828 +0000 UTC m=+1164.853426605" watchObservedRunningTime="2026-09-09 12:30:42.225206678 +0000 UTC m=+1164.855794451"
Sep 09 12:30:48.176441 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:48.176411    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:30:48.176872 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:48.176696    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:30:51.512035 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:51.511987    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:51.512663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:51.512048    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:51.524694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:51.524670    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:52.260002 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:52.259972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:30:57.748220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:57.748185    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:30:57.748725 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:57.748421    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" podUID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" containerName="payload-processing" containerID="cri-o://2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51" gracePeriod=30
Sep 09 12:30:57.759491 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:57.759455    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:30:57.759867 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:57.759695    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" podUID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" containerName="payload-pre-processing" containerID="cri-o://76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2" gracePeriod=30
Sep 09 12:30:58.039455 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.039432    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:30:58.042041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.042021    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:30:58.062903 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.062875    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume\") pod \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") "
Sep 09 12:30:58.063052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.062926    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpbzb\" (UniqueName: \"kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb\") pod \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") "
Sep 09 12:30:58.063052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.062953    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume\") pod \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\" (UID: \"66c6a4a3-701e-44bf-b781-dfed2f9c27e8\") "
Sep 09 12:30:58.063052 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.062991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgnf7\" (UniqueName: \"kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7\") pod \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\" (UID: \"003e846b-e79f-4e92-9dd2-f73e46c4eb9c\") "
Sep 09 12:30:58.063471 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.063444    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "003e846b-e79f-4e92-9dd2-f73e46c4eb9c" (UID: "003e846b-e79f-4e92-9dd2-f73e46c4eb9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:30:58.063578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.063474    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66c6a4a3-701e-44bf-b781-dfed2f9c27e8" (UID: "66c6a4a3-701e-44bf-b781-dfed2f9c27e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:30:58.065358 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.065329    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb" (OuterVolumeSpecName: "kube-api-access-zpbzb") pod "66c6a4a3-701e-44bf-b781-dfed2f9c27e8" (UID: "66c6a4a3-701e-44bf-b781-dfed2f9c27e8"). InnerVolumeSpecName "kube-api-access-zpbzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:30:58.065476 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.065455    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7" (OuterVolumeSpecName: "kube-api-access-lgnf7") pod "003e846b-e79f-4e92-9dd2-f73e46c4eb9c" (UID: "003e846b-e79f-4e92-9dd2-f73e46c4eb9c"). InnerVolumeSpecName "kube-api-access-lgnf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:30:58.164639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.164603    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbzb\" (UniqueName: \"kubernetes.io/projected/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-kube-api-access-zpbzb\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:30:58.164639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.164631    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c6a4a3-701e-44bf-b781-dfed2f9c27e8-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:30:58.164639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.164641    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnf7\" (UniqueName: \"kubernetes.io/projected/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-kube-api-access-lgnf7\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:30:58.164639 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.164650    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e846b-e79f-4e92-9dd2-f73e46c4eb9c-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:30:58.281685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.281600    2556 generic.go:356] "Generic (PLEG): container finished" podID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" containerID="2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51" exitCode=0
Sep 09 12:30:58.281685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.281648    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"
Sep 09 12:30:58.281685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.281671    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" event={"ID":"66c6a4a3-701e-44bf-b781-dfed2f9c27e8","Type":"ContainerDied","Data":"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"}
Sep 09 12:30:58.281939 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.281734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx" event={"ID":"66c6a4a3-701e-44bf-b781-dfed2f9c27e8","Type":"ContainerDied","Data":"e6595ca2ee9c9c7e97328a09f7f2bf188079b307c9f9ffc173f4b8162778a84a"}
Sep 09 12:30:58.281939 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.281765    2556 scope.go:117] "RemoveContainer" containerID="2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"
Sep 09 12:30:58.283072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.283050    2556 generic.go:356] "Generic (PLEG): container finished" podID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" containerID="76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2" exitCode=0
Sep 09 12:30:58.283182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.283092    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"
Sep 09 12:30:58.283182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.283124    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" event={"ID":"003e846b-e79f-4e92-9dd2-f73e46c4eb9c","Type":"ContainerDied","Data":"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"}
Sep 09 12:30:58.283182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.283162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2" event={"ID":"003e846b-e79f-4e92-9dd2-f73e46c4eb9c","Type":"ContainerDied","Data":"91b8f79741ef11e57daefb8a70d9b6481e9f49cd95c1f768cd04ae3765b326b6"}
Sep 09 12:30:58.294426 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.294396    2556 scope.go:117] "RemoveContainer" containerID="2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"
Sep 09 12:30:58.294749 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:30:58.294726    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51\": container with ID starting with 2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51 not found: ID does not exist" containerID="2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"
Sep 09 12:30:58.294847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.294755    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51"} err="failed to get container status \"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51\": rpc error: code = NotFound desc = could not find container \"2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51\": container with ID starting with 2bb40e880dd0e01f4f5a8467b4036d65eb4e7265dbdbef07ce3da1fc7f94ec51 not found: ID does not exist"
Sep 09 12:30:58.294847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.294773    2556 scope.go:117] "RemoveContainer" containerID="76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"
Sep 09 12:30:58.303206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.303189    2556 scope.go:117] "RemoveContainer" containerID="76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"
Sep 09 12:30:58.303498 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:30:58.303481    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2\": container with ID starting with 76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2 not found: ID does not exist" containerID="76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"
Sep 09 12:30:58.303547 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.303505    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2"} err="failed to get container status \"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2\": rpc error: code = NotFound desc = could not find container \"76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2\": container with ID starting with 76b20ec39afeec0daea1d7efa86fb57242968f5038a3c8228acb7721e944e6e2 not found: ID does not exist"
Sep 09 12:30:58.310347 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.310324    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:30:58.314401 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.314381    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e49066-6f8456d674-t8lt2"]
Sep 09 12:30:58.329477 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.329454    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:30:58.342982 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:58.342955    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e49066-667c97cb85-w4rwx"]
Sep 09 12:30:59.594305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594270    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:30:59.594670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594622    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" containerName="payload-processing"
Sep 09 12:30:59.594670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594634    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" containerName="payload-processing"
Sep 09 12:30:59.594670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594654    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" containerName="payload-pre-processing"
Sep 09 12:30:59.594670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594660    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" containerName="payload-pre-processing"
Sep 09 12:30:59.594812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594715    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" containerName="payload-pre-processing"
Sep 09 12:30:59.594812 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.594724    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" containerName="payload-processing"
Sep 09 12:30:59.599148 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.599125    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:30:59.601653 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.601630    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kwwl8\""
Sep 09 12:30:59.610845 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.610818    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:30:59.677669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.677623    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgt2\" (UniqueName: \"kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2\") pod \"test-curl-7881-139803507677776\" (UID: \"47719705-3b0a-427f-bcf5-bb9a86a4038e\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:30:59.778805 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.778768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgt2\" (UniqueName: \"kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2\") pod \"test-curl-7881-139803507677776\" (UID: \"47719705-3b0a-427f-bcf5-bb9a86a4038e\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:30:59.790669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.790639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgt2\" (UniqueName: \"kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2\") pod \"test-curl-7881-139803507677776\" (UID: \"47719705-3b0a-427f-bcf5-bb9a86a4038e\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:30:59.908537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.904032    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003e846b-e79f-4e92-9dd2-f73e46c4eb9c" path="/var/lib/kubelet/pods/003e846b-e79f-4e92-9dd2-f73e46c4eb9c/volumes"
Sep 09 12:30:59.908537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.904635    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c6a4a3-701e-44bf-b781-dfed2f9c27e8" path="/var/lib/kubelet/pods/66c6a4a3-701e-44bf-b781-dfed2f9c27e8/volumes"
Sep 09 12:30:59.908537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:30:59.908313    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:00.185763 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:00.185739    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:00.187791 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:00.187762    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47719705_3b0a_427f_bcf5_bb9a86a4038e.slice/crio-11c77145e0a70d9e3de318b822cce973bf1996ee48cee80efd901d29d6ceefc9 WatchSource:0}: Error finding container 11c77145e0a70d9e3de318b822cce973bf1996ee48cee80efd901d29d6ceefc9: Status 404 returned error can't find the container with id 11c77145e0a70d9e3de318b822cce973bf1996ee48cee80efd901d29d6ceefc9
Sep 09 12:31:00.296966 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:00.296931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803507677776" event={"ID":"47719705-3b0a-427f-bcf5-bb9a86a4038e","Type":"ContainerStarted","Data":"11c77145e0a70d9e3de318b822cce973bf1996ee48cee80efd901d29d6ceefc9"}
Sep 09 12:31:02.308650 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:02.308615    2556 generic.go:356] "Generic (PLEG): container finished" podID="47719705-3b0a-427f-bcf5-bb9a86a4038e" containerID="bdf38a155817ceff03f9929983669603d1d1c47cf544a0814c5d16aea457c347" exitCode=0
Sep 09 12:31:02.309105 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:02.308670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803507677776" event={"ID":"47719705-3b0a-427f-bcf5-bb9a86a4038e","Type":"ContainerDied","Data":"bdf38a155817ceff03f9929983669603d1d1c47cf544a0814c5d16aea457c347"}
Sep 09 12:31:03.453584 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.453556    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:03.500710 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.500663    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7881-139803507677776_47719705-3b0a-427f-bcf5-bb9a86a4038e/test-curl-7881-139803507677776/0.log"
Sep 09 12:31:03.513967 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.513930    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljgt2\" (UniqueName: \"kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2\") pod \"47719705-3b0a-427f-bcf5-bb9a86a4038e\" (UID: \"47719705-3b0a-427f-bcf5-bb9a86a4038e\") "
Sep 09 12:31:03.516174 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.516148    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2" (OuterVolumeSpecName: "kube-api-access-ljgt2") pod "47719705-3b0a-427f-bcf5-bb9a86a4038e" (UID: "47719705-3b0a-427f-bcf5-bb9a86a4038e"). InnerVolumeSpecName "kube-api-access-ljgt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:03.536659 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.536628    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:03.542785 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.542759    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:03.615694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.615612    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgt2\" (UniqueName: \"kubernetes.io/projected/47719705-3b0a-427f-bcf5-bb9a86a4038e-kube-api-access-ljgt2\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:03.668428 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.668389    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:03.668819 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.668800    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47719705-3b0a-427f-bcf5-bb9a86a4038e" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:03.668915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.668823    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47719705-3b0a-427f-bcf5-bb9a86a4038e" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:03.668967 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.668957    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47719705-3b0a-427f-bcf5-bb9a86a4038e" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:03.672150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.672130    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:03.686292 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.686265    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:03.716923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.716882    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrvn\" (UniqueName: \"kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn\") pod \"test-curl-7881-139803507677776\" (UID: \"195cf3a8-36d3-4339-8fa1-c4b767ac1971\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:03.817694 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.817652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrvn\" (UniqueName: \"kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn\") pod \"test-curl-7881-139803507677776\" (UID: \"195cf3a8-36d3-4339-8fa1-c4b767ac1971\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:03.827939 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.827898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrvn\" (UniqueName: \"kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn\") pod \"test-curl-7881-139803507677776\" (UID: \"195cf3a8-36d3-4339-8fa1-c4b767ac1971\") " pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:03.902140 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.902056    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47719705-3b0a-427f-bcf5-bb9a86a4038e" path="/var/lib/kubelet/pods/47719705-3b0a-427f-bcf5-bb9a86a4038e/volumes"
Sep 09 12:31:03.981224 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:03.981192    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:04.256295 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:04.256259    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:04.259396 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:04.259364    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod195cf3a8_36d3_4339_8fa1_c4b767ac1971.slice/crio-bbcc2bf54dbf88373876116542c899797a99f5888a673e9688e941fcc779630e WatchSource:0}: Error finding container bbcc2bf54dbf88373876116542c899797a99f5888a673e9688e941fcc779630e: Status 404 returned error can't find the container with id bbcc2bf54dbf88373876116542c899797a99f5888a673e9688e941fcc779630e
Sep 09 12:31:04.321342 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:04.321304    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803507677776" event={"ID":"195cf3a8-36d3-4339-8fa1-c4b767ac1971","Type":"ContainerStarted","Data":"bbcc2bf54dbf88373876116542c899797a99f5888a673e9688e941fcc779630e"}
Sep 09 12:31:04.322575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:04.322548    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:04.322719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:04.322553    2556 scope.go:117] "RemoveContainer" containerID="bdf38a155817ceff03f9929983669603d1d1c47cf544a0814c5d16aea457c347"
Sep 09 12:31:06.335115 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:06.335081    2556 generic.go:356] "Generic (PLEG): container finished" podID="195cf3a8-36d3-4339-8fa1-c4b767ac1971" containerID="4ae97fa44bc474059798c68eab486d8fa8c667c89c6bd40bf7670223c30c8f09" exitCode=0
Sep 09 12:31:06.335519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:06.335164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803507677776" event={"ID":"195cf3a8-36d3-4339-8fa1-c4b767ac1971","Type":"ContainerDied","Data":"4ae97fa44bc474059798c68eab486d8fa8c667c89c6bd40bf7670223c30c8f09"}
Sep 09 12:31:07.544543 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.544516    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:07.568149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.568117    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7881-139803507677776_195cf3a8-36d3-4339-8fa1-c4b767ac1971/test-curl-7881-139803507677776/0.log"
Sep 09 12:31:07.599874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.599790    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:07.607587 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.607555    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-139803507677776"]
Sep 09 12:31:07.663297 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.663231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcrvn\" (UniqueName: \"kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn\") pod \"195cf3a8-36d3-4339-8fa1-c4b767ac1971\" (UID: \"195cf3a8-36d3-4339-8fa1-c4b767ac1971\") "
Sep 09 12:31:07.665618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.665574    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn" (OuterVolumeSpecName: "kube-api-access-gcrvn") pod "195cf3a8-36d3-4339-8fa1-c4b767ac1971" (UID: "195cf3a8-36d3-4339-8fa1-c4b767ac1971"). InnerVolumeSpecName "kube-api-access-gcrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:07.727962 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.727929    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:07.728333 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.728320    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195cf3a8-36d3-4339-8fa1-c4b767ac1971" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:07.728384 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.728335    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="195cf3a8-36d3-4339-8fa1-c4b767ac1971" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:07.728422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.728412    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="195cf3a8-36d3-4339-8fa1-c4b767ac1971" containerName="test-curl-7881-139803507677776"
Sep 09 12:31:07.731612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.731584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:07.743150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.743113    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:07.764428 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.764390    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrvn\" (UniqueName: \"kubernetes.io/projected/195cf3a8-36d3-4339-8fa1-c4b767ac1971-kube-api-access-gcrvn\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:07.865747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.865645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c74\" (UniqueName: \"kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74\") pod \"test-curl-7881-139803509895648\" (UID: \"6a46488c-97e1-41a0-a67a-13ec4b3125f6\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:07.902951 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.902915    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195cf3a8-36d3-4339-8fa1-c4b767ac1971" path="/var/lib/kubelet/pods/195cf3a8-36d3-4339-8fa1-c4b767ac1971/volumes"
Sep 09 12:31:07.966592 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.966560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c74\" (UniqueName: \"kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74\") pod \"test-curl-7881-139803509895648\" (UID: \"6a46488c-97e1-41a0-a67a-13ec4b3125f6\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:07.978296 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:07.978268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c74\" (UniqueName: \"kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74\") pod \"test-curl-7881-139803509895648\" (UID: \"6a46488c-97e1-41a0-a67a-13ec4b3125f6\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:08.042313 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:08.042231    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:08.322192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:08.322168    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:08.324347 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:08.324315    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a46488c_97e1_41a0_a67a_13ec4b3125f6.slice/crio-3be9309b4410adef8da4d623e86e89a9e32415a9d78b344868a99c0a8a32bd72 WatchSource:0}: Error finding container 3be9309b4410adef8da4d623e86e89a9e32415a9d78b344868a99c0a8a32bd72: Status 404 returned error can't find the container with id 3be9309b4410adef8da4d623e86e89a9e32415a9d78b344868a99c0a8a32bd72
Sep 09 12:31:08.347859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:08.347819    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803509895648" event={"ID":"6a46488c-97e1-41a0-a67a-13ec4b3125f6","Type":"ContainerStarted","Data":"3be9309b4410adef8da4d623e86e89a9e32415a9d78b344868a99c0a8a32bd72"}
Sep 09 12:31:08.349206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:08.349180    2556 scope.go:117] "RemoveContainer" containerID="4ae97fa44bc474059798c68eab486d8fa8c667c89c6bd40bf7670223c30c8f09"
Sep 09 12:31:08.349206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:08.349191    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803507677776"
Sep 09 12:31:09.357071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:09.357035    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" containerID="49ce5dac57b1eec2b03e49b131f43122ee248335b78ae0b57b55f03e5b8c8dda" exitCode=0
Sep 09 12:31:09.357542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:09.357126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803509895648" event={"ID":"6a46488c-97e1-41a0-a67a-13ec4b3125f6","Type":"ContainerDied","Data":"49ce5dac57b1eec2b03e49b131f43122ee248335b78ae0b57b55f03e5b8c8dda"}
Sep 09 12:31:10.498841 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.498813    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:10.529715 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.529686    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7881-139803509895648_6a46488c-97e1-41a0-a67a-13ec4b3125f6/test-curl-7881-139803509895648/0.log"
Sep 09 12:31:10.561324 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.561287    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:10.566802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.566765    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:10.592809 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.592780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86c74\" (UniqueName: \"kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74\") pod \"6a46488c-97e1-41a0-a67a-13ec4b3125f6\" (UID: \"6a46488c-97e1-41a0-a67a-13ec4b3125f6\") "
Sep 09 12:31:10.595028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.594996    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74" (OuterVolumeSpecName: "kube-api-access-86c74") pod "6a46488c-97e1-41a0-a67a-13ec4b3125f6" (UID: "6a46488c-97e1-41a0-a67a-13ec4b3125f6"). InnerVolumeSpecName "kube-api-access-86c74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:10.690364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.690280    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:10.690659 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.690646    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:10.690697 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.690660    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:10.690739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.690730    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:10.693643 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.693622    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:10.693848 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.693824    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c74\" (UniqueName: \"kubernetes.io/projected/6a46488c-97e1-41a0-a67a-13ec4b3125f6-kube-api-access-86c74\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:10.702271 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.702220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:10.794613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.794572    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jj9\" (UniqueName: \"kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9\") pod \"test-curl-7881-139803509895648\" (UID: \"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:10.895811 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.895779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jj9\" (UniqueName: \"kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9\") pod \"test-curl-7881-139803509895648\" (UID: \"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:10.909448 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:10.909414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jj9\" (UniqueName: \"kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9\") pod \"test-curl-7881-139803509895648\" (UID: \"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0\") " pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:11.003465 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.003426    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:11.276614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.276589    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:11.278679 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:11.278654    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec1ebf93_2a69_46a7_99bf_4c8094ba44a0.slice/crio-0971b96eadef432c724d5d34d8ce2791cf5b9444a3927da01a77c7b81622403f WatchSource:0}: Error finding container 0971b96eadef432c724d5d34d8ce2791cf5b9444a3927da01a77c7b81622403f: Status 404 returned error can't find the container with id 0971b96eadef432c724d5d34d8ce2791cf5b9444a3927da01a77c7b81622403f
Sep 09 12:31:11.370068 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.370026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803509895648" event={"ID":"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0","Type":"ContainerStarted","Data":"0971b96eadef432c724d5d34d8ce2791cf5b9444a3927da01a77c7b81622403f"}
Sep 09 12:31:11.371353 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.371336    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be9309b4410adef8da4d623e86e89a9e32415a9d78b344868a99c0a8a32bd72"
Sep 09 12:31:11.371437 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.371371    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:11.383439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.383408    2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7881-139803509895648" oldPodUID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" podUID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0"
Sep 09 12:31:11.902410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:11.902381    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a46488c-97e1-41a0-a67a-13ec4b3125f6" path="/var/lib/kubelet/pods/6a46488c-97e1-41a0-a67a-13ec4b3125f6/volumes"
Sep 09 12:31:12.378032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:12.377995    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" containerID="8573b78262a1b994e6f1ec305d514e5420615c073059b78c0b19654a4df853ab" exitCode=0
Sep 09 12:31:12.378217 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:12.378051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-139803509895648" event={"ID":"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0","Type":"ContainerDied","Data":"8573b78262a1b994e6f1ec305d514e5420615c073059b78c0b19654a4df853ab"}
Sep 09 12:31:13.522038 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.522009    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:13.547328 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.547296    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7881-139803509895648_ec1ebf93-2a69-46a7-99bf-4c8094ba44a0/test-curl-7881-139803509895648/0.log"
Sep 09 12:31:13.575657 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.575622    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:13.586542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.586511    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-139803509895648"]
Sep 09 12:31:13.620531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.620490    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jj9\" (UniqueName: \"kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9\") pod \"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0\" (UID: \"ec1ebf93-2a69-46a7-99bf-4c8094ba44a0\") "
Sep 09 12:31:13.622693 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.622665    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9" (OuterVolumeSpecName: "kube-api-access-97jj9") pod "ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" (UID: "ec1ebf93-2a69-46a7-99bf-4c8094ba44a0"). InnerVolumeSpecName "kube-api-access-97jj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:13.721358 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.721327    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jj9\" (UniqueName: \"kubernetes.io/projected/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0-kube-api-access-97jj9\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:13.905380 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:13.903749    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" path="/var/lib/kubelet/pods/ec1ebf93-2a69-46a7-99bf-4c8094ba44a0/volumes"
Sep 09 12:31:14.390014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:14.389977    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-139803509895648"
Sep 09 12:31:14.390014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:14.390001    2556 scope.go:117] "RemoveContainer" containerID="8573b78262a1b994e6f1ec305d514e5420615c073059b78c0b19654a4df853ab"
Sep 09 12:31:17.851094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:17.851060    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:31:17.853190 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:17.853162    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:31:17.855514 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:17.855490    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:31:17.857344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:17.857326    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:31:18.060523 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:18.060490    2556 scope.go:117] "RemoveContainer" containerID="b8862c83b023845cd63063ee3467e259d4d51b403806a38dd3fdfaf63ed0b66d"
Sep 09 12:31:19.971086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:19.971047    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:31:20.455368 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.455291    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:31:20.455663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.455651    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:20.455712 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.455664    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:20.455753 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.455743    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1ebf93-2a69-46a7-99bf-4c8094ba44a0" containerName="test-curl-7881-139803509895648"
Sep 09 12:31:20.459174 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.459156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.461733 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.461710    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-24ec44a5-dockercfg-lx4g4\""
Sep 09 12:31:20.461839 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.461767    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-24ec44a5\""
Sep 09 12:31:20.471273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.471229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:31:20.517637 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.517603    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:31:20.520647 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.520626    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.535918 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.535888    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:31:20.584837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.584807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.585000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.584845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzsp\" (UniqueName: \"kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.686013 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.685982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwc5\" (UniqueName: \"kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.686194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.686043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.686194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.686082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.686194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.686101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzsp\" (UniqueName: \"kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.686813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.686786    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.702921 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.702882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzsp\" (UniqueName: \"kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp\") pod \"payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.769109 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.769068    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:20.786841 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.786814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.786936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.786898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwc5\" (UniqueName: \"kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.787733 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.787708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.796609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.796579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwc5\" (UniqueName: \"kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5\") pod \"payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") " pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:20.830422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:20.830392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:21.051156 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.051129    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:31:21.052950 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:21.052912    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6f1fb7_7094_4bab_b115_87a6aa463596.slice/crio-f225626e39b063c8949a70bf548501e5daef7b70eedfe40b2c8ed2d6c8cc10c8 WatchSource:0}: Error finding container f225626e39b063c8949a70bf548501e5daef7b70eedfe40b2c8ed2d6c8cc10c8: Status 404 returned error can't find the container with id f225626e39b063c8949a70bf548501e5daef7b70eedfe40b2c8ed2d6c8cc10c8
Sep 09 12:31:21.109277 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.109252    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:31:21.111178 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:31:21.111151    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455b70d5_b242_4684_b988_dafa469b1e0c.slice/crio-e815364a4aa95bf1c22b65f914ca49d2c298a5309a317a0ac5a33c94e0f44315 WatchSource:0}: Error finding container e815364a4aa95bf1c22b65f914ca49d2c298a5309a317a0ac5a33c94e0f44315: Status 404 returned error can't find the container with id e815364a4aa95bf1c22b65f914ca49d2c298a5309a317a0ac5a33c94e0f44315
Sep 09 12:31:21.429407 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.429315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" event={"ID":"455b70d5-b242-4684-b988-dafa469b1e0c","Type":"ContainerStarted","Data":"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"}
Sep 09 12:31:21.429407 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.429372    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" event={"ID":"455b70d5-b242-4684-b988-dafa469b1e0c","Type":"ContainerStarted","Data":"e815364a4aa95bf1c22b65f914ca49d2c298a5309a317a0ac5a33c94e0f44315"}
Sep 09 12:31:21.429407 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.429407    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:21.431025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.431002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" event={"ID":"5b6f1fb7-7094-4bab-b115-87a6aa463596","Type":"ContainerStarted","Data":"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"}
Sep 09 12:31:21.431145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.431042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" event={"ID":"5b6f1fb7-7094-4bab-b115-87a6aa463596","Type":"ContainerStarted","Data":"f225626e39b063c8949a70bf548501e5daef7b70eedfe40b2c8ed2d6c8cc10c8"}
Sep 09 12:31:21.431145 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.431091    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:21.451696 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.451655    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" podStartSLOduration=1.4516411009999999 podStartE2EDuration="1.451641101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:31:21.451415964 +0000 UTC m=+1204.082003739" watchObservedRunningTime="2026-09-09 12:31:21.451641101 +0000 UTC m=+1204.082228877"
Sep 09 12:31:21.470406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:21.470363    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" podStartSLOduration=1.470348878 podStartE2EDuration="1.470348878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:31:21.469374418 +0000 UTC m=+1204.099962193" watchObservedRunningTime="2026-09-09 12:31:21.470348878 +0000 UTC m=+1204.100936656"
Sep 09 12:31:32.438296 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:32.438255    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:31:32.438731 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:32.438384    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:31:34.991362 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:34.991319    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" podUID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" containerName="main" containerID="cri-o://f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8" gracePeriod=45
Sep 09 12:31:35.238658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.238628    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:31:35.326107 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326024    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326107 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326082    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326145    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326177    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326207    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326265    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w79q7\" (UniqueName: \"kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7\") pod \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\" (UID: \"927f1a60-16dd-4864-a836-b40eb0eb7aa8\") "
Sep 09 12:31:35.326352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326283    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home" (OuterVolumeSpecName: "home") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:31:35.326616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326408    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:31:35.326616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326424    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache" (OuterVolumeSpecName: "model-cache") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:31:35.326616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326605    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.326719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326631    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.326719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.326649    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.328498 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.328466    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:31:35.328625 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.328562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7" (OuterVolumeSpecName: "kube-api-access-w79q7") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "kube-api-access-w79q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:35.328625 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.328560    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm" (OuterVolumeSpecName: "dshm") pod "927f1a60-16dd-4864-a836-b40eb0eb7aa8" (UID: "927f1a60-16dd-4864-a836-b40eb0eb7aa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:31:35.427414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.427385    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/927f1a60-16dd-4864-a836-b40eb0eb7aa8-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.427414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.427411    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/927f1a60-16dd-4864-a836-b40eb0eb7aa8-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.427414 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.427421    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79q7\" (UniqueName: \"kubernetes.io/projected/927f1a60-16dd-4864-a836-b40eb0eb7aa8-kube-api-access-w79q7\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:35.512844 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.512809    2556 generic.go:356] "Generic (PLEG): container finished" podID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" containerID="f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8" exitCode=0
Sep 09 12:31:35.513039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.512863    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"
Sep 09 12:31:35.513039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.512895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" event={"ID":"927f1a60-16dd-4864-a836-b40eb0eb7aa8","Type":"ContainerDied","Data":"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"}
Sep 09 12:31:35.513039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.512946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw" event={"ID":"927f1a60-16dd-4864-a836-b40eb0eb7aa8","Type":"ContainerDied","Data":"19896de72f78c2fc0c7d22226b8bacdd792ab5617889db40d6b4c5fef9ad6f9b"}
Sep 09 12:31:35.513039 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.512970    2556 scope.go:117] "RemoveContainer" containerID="f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"
Sep 09 12:31:35.522417 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.522400    2556 scope.go:117] "RemoveContainer" containerID="f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"
Sep 09 12:31:35.522665 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:31:35.522649    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8\": container with ID starting with f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8 not found: ID does not exist" containerID="f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"
Sep 09 12:31:35.522718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.522672    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8"} err="failed to get container status \"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8\": rpc error: code = NotFound desc = could not find container \"f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8\": container with ID starting with f33d1168526a0c65761aa0be898d2d3567021d9eadad91adedec1039f747d2e8 not found: ID does not exist"
Sep 09 12:31:35.538853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.538831    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:31:35.546635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.546616    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-db67bc/noten-model-db67bc-kserve-7cb65f4795-c5rpw"]
Sep 09 12:31:35.901948 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:35.901917    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" path="/var/lib/kubelet/pods/927f1a60-16dd-4864-a836-b40eb0eb7aa8/volumes"
Sep 09 12:31:52.502382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.502350    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"]
Sep 09 12:31:52.513035 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.513005    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a142a-29815950-n95mw"]
Sep 09 12:31:52.559587 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.559555    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:31:52.559785 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.559762    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" podUID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" containerName="payload-pre-processing" containerID="cri-o://3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6" gracePeriod=30
Sep 09 12:31:52.815635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.815601    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:31:52.890665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.890615    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn7w2\" (UniqueName: \"kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2\") pod \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") "
Sep 09 12:31:52.890665 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.890666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume\") pod \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\" (UID: \"c31a81d0-0aa4-43cb-9fda-cc80d114c936\") "
Sep 09 12:31:52.891021 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.891002    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c31a81d0-0aa4-43cb-9fda-cc80d114c936" (UID: "c31a81d0-0aa4-43cb-9fda-cc80d114c936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:31:52.892890 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.892857    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2" (OuterVolumeSpecName: "kube-api-access-wn7w2") pod "c31a81d0-0aa4-43cb-9fda-cc80d114c936" (UID: "c31a81d0-0aa4-43cb-9fda-cc80d114c936"). InnerVolumeSpecName "kube-api-access-wn7w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:31:52.992288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.992207    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7w2\" (UniqueName: \"kubernetes.io/projected/c31a81d0-0aa4-43cb-9fda-cc80d114c936-kube-api-access-wn7w2\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:52.992288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:52.992284    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31a81d0-0aa4-43cb-9fda-cc80d114c936-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:31:53.607164 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.607125    2556 generic.go:356] "Generic (PLEG): container finished" podID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" containerID="3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6" exitCode=0
Sep 09 12:31:53.607624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.607171    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"
Sep 09 12:31:53.607624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.607187    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" event={"ID":"c31a81d0-0aa4-43cb-9fda-cc80d114c936","Type":"ContainerDied","Data":"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"}
Sep 09 12:31:53.607624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.607265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9" event={"ID":"c31a81d0-0aa4-43cb-9fda-cc80d114c936","Type":"ContainerDied","Data":"318e5c2d41f5eea916c6b0a04e97d0bc739260fcfa4a34e8b6ffd033a5ff8e9d"}
Sep 09 12:31:53.607624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.607297    2556 scope.go:117] "RemoveContainer" containerID="3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"
Sep 09 12:31:53.616451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.616433    2556 scope.go:117] "RemoveContainer" containerID="3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"
Sep 09 12:31:53.616764 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:31:53.616744    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6\": container with ID starting with 3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6 not found: ID does not exist" containerID="3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"
Sep 09 12:31:53.616813 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.616773    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6"} err="failed to get container status \"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6\": rpc error: code = NotFound desc = could not find container \"3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6\": container with ID starting with 3f47c7928fa942d639d801814b72b8afcffec7f61c308bdd3f9c91662860a1b6 not found: ID does not exist"
Sep 09 12:31:53.632133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.632112    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:31:53.637959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.637935    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a142a-7ddb49fccd-79vt9"]
Sep 09 12:31:53.901961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.901883    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6557f0-0ca8-4766-afff-2f555904a3aa" path="/var/lib/kubelet/pods/9f6557f0-0ca8-4766-afff-2f555904a3aa/volumes"
Sep 09 12:31:53.902195 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:31:53.902182    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" path="/var/lib/kubelet/pods/c31a81d0-0aa4-43cb-9fda-cc80d114c936/volumes"
Sep 09 12:32:06.583737 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.583704    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584060    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" containerName="main"
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584072    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" containerName="main"
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584106    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" containerName="payload-pre-processing"
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584111    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" containerName="payload-pre-processing"
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584183    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31a81d0-0aa4-43cb-9fda-cc80d114c936" containerName="payload-pre-processing"
Sep 09 12:32:06.586069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.584191    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="927f1a60-16dd-4864-a836-b40eb0eb7aa8" containerName="main"
Sep 09 12:32:06.587086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.587071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.589537 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.589513    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-679f45\""
Sep 09 12:32:06.589651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.589589    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-679f45-dockercfg-9w585\""
Sep 09 12:32:06.598676 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.598655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:32:06.711571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.711537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7x2\" (UniqueName: \"kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.711746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.711607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.812902 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.812870    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7x2\" (UniqueName: \"kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.813070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.812927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.813576 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.813557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.823901 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.823878    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7x2\" (UniqueName: \"kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2\") pod \"payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:06.897421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:06.897335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:07.182635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:07.182610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:32:07.184601 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:32:07.184573    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990fb1c7_d48d_4d73_a8c9_49fc5f4574f0.slice/crio-05cdccfed528861a6386499226209a3d594ccd39d0c7bbcd57553692916c5bc6 WatchSource:0}: Error finding container 05cdccfed528861a6386499226209a3d594ccd39d0c7bbcd57553692916c5bc6: Status 404 returned error can't find the container with id 05cdccfed528861a6386499226209a3d594ccd39d0c7bbcd57553692916c5bc6
Sep 09 12:32:07.683597 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:07.683561    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" event={"ID":"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0","Type":"ContainerStarted","Data":"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"}
Sep 09 12:32:07.684023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:07.683621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" event={"ID":"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0","Type":"ContainerStarted","Data":"05cdccfed528861a6386499226209a3d594ccd39d0c7bbcd57553692916c5bc6"}
Sep 09 12:32:07.684023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:07.683650    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:07.702877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:07.702828    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" podStartSLOduration=1.7028139169999998 podStartE2EDuration="1.702813917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:32:07.700471265 +0000 UTC m=+1250.331059040" watchObservedRunningTime="2026-09-09 12:32:07.702813917 +0000 UTC m=+1250.333401693"
Sep 09 12:32:18.691276 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:18.691221    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:32:32.005379 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.005343    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:32:32.010513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.010478    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.015320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.015300    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-24ec44a5\"/\"kube-root-ca.crt\""
Sep 09 12:32:32.015513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.015496    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-24ec44a5\"/\"default-dockercfg-qrsc8\""
Sep 09 12:32:32.016273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.016251    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-24ec44a5\"/\"openshift-service-ca.crt\""
Sep 09 12:32:32.016391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.016373    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-24ec44a5\"/\"e2e-lifecycle-model-24ec44a5-kserve-self-signed-certs\""
Sep 09 12:32:32.053206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.053173    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:32:32.056891 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.056858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46vr\" (UniqueName: \"kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.057012 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.056946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.057012 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.056978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.057138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.057018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.057138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.057040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.057138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.057106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46vr\" (UniqueName: \"kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158846    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.158999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.158957    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.160900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.160882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.161173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.161155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.169763 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.169743    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46vr\" (UniqueName: \"kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr\") pod \"e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") " pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.319566 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.319477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:32.596747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.596708    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:32:32.597474 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:32:32.597414    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c43429c_e9a7_4d4b_9429_6703e9a37669.slice/crio-b887ad84b11544dce5a02bf1f39cf47c4e59dacbf78c7f918a4552378201dac4 WatchSource:0}: Error finding container b887ad84b11544dce5a02bf1f39cf47c4e59dacbf78c7f918a4552378201dac4: Status 404 returned error can't find the container with id b887ad84b11544dce5a02bf1f39cf47c4e59dacbf78c7f918a4552378201dac4
Sep 09 12:32:32.810036 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.809998    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" event={"ID":"9c43429c-e9a7-4d4b-9429-6703e9a37669","Type":"ContainerStarted","Data":"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"}
Sep 09 12:32:32.810286 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.810046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" event={"ID":"9c43429c-e9a7-4d4b-9429-6703e9a37669","Type":"ContainerStarted","Data":"b887ad84b11544dce5a02bf1f39cf47c4e59dacbf78c7f918a4552378201dac4"}
Sep 09 12:32:32.833370 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:32.833323    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" podStartSLOduration=1.83330862 podStartE2EDuration="1.83330862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:32:32.831806325 +0000 UTC m=+1275.462394102" watchObservedRunningTime="2026-09-09 12:32:32.83330862 +0000 UTC m=+1275.463896427"
Sep 09 12:32:35.708578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.708539    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:32:35.712356 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.712335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.715131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.715103    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-cfcdfd-dockercfg-lbpnj\""
Sep 09 12:32:35.715399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.715375    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-cfcdfd\""
Sep 09 12:32:35.724399 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.724377    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:32:35.726674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.726655    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:32:35.730229 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.730211    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.741726 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.741700    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:32:35.793032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.793000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvnj\" (UniqueName: \"kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.793208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.793057    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9b6\" (UniqueName: \"kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.793208 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.793173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.793336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.793215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.893724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.893690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvnj\" (UniqueName: \"kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.893895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.893740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9b6\" (UniqueName: \"kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.893938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.893904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.893978 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.893956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.894868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.894847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:35.894915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.894852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.905028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.904999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9b6\" (UniqueName: \"kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6\") pod \"payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:35.905513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:35.905495    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvnj\" (UniqueName: \"kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj\") pod \"payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:36.024872 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.024834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:36.042073 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.042047    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:36.309196 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.309168    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:32:36.312394 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:32:36.312363    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3852d1_dcc4_4df3_8c9e_2944f477a0fa.slice/crio-71d79b80e2208b6b64c8033063d2699254d6cc6b758ad62b32190b6190ec82ad WatchSource:0}: Error finding container 71d79b80e2208b6b64c8033063d2699254d6cc6b758ad62b32190b6190ec82ad: Status 404 returned error can't find the container with id 71d79b80e2208b6b64c8033063d2699254d6cc6b758ad62b32190b6190ec82ad
Sep 09 12:32:36.326072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.326051    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:32:36.328231 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:32:36.328201    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0856ffa0_8108_4e71_be97_630e2b5ec701.slice/crio-5c3f50c748be2590214687e2931c99301f381804bd2b5b7cb3e4e82980f89f61 WatchSource:0}: Error finding container 5c3f50c748be2590214687e2931c99301f381804bd2b5b7cb3e4e82980f89f61: Status 404 returned error can't find the container with id 5c3f50c748be2590214687e2931c99301f381804bd2b5b7cb3e4e82980f89f61
Sep 09 12:32:36.831543 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.831505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" event={"ID":"0856ffa0-8108-4e71-be97-630e2b5ec701","Type":"ContainerStarted","Data":"6c5fe01d664c56c5f640b16ebce830de3c2c15c009b4ee9ec26ee0e5de94df33"}
Sep 09 12:32:36.832014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.831556    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" event={"ID":"0856ffa0-8108-4e71-be97-630e2b5ec701","Type":"ContainerStarted","Data":"5c3f50c748be2590214687e2931c99301f381804bd2b5b7cb3e4e82980f89f61"}
Sep 09 12:32:36.832014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.831676    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:36.833118 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.833095    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" event={"ID":"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa","Type":"ContainerStarted","Data":"95309e9a1d5dc8f65870596958e4ff7e93ecfff72f04c20e4ede0a73b9c037a3"}
Sep 09 12:32:36.833302 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.833132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" event={"ID":"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa","Type":"ContainerStarted","Data":"71d79b80e2208b6b64c8033063d2699254d6cc6b758ad62b32190b6190ec82ad"}
Sep 09 12:32:36.833302 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.833150    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:32:36.853280 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.853210    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" podStartSLOduration=1.853196179 podStartE2EDuration="1.853196179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:32:36.851498069 +0000 UTC m=+1279.482085845" watchObservedRunningTime="2026-09-09 12:32:36.853196179 +0000 UTC m=+1279.483783951"
Sep 09 12:32:36.878084 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:36.878032    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" podStartSLOduration=1.87801691 podStartE2EDuration="1.87801691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:32:36.877124747 +0000 UTC m=+1279.507712522" watchObservedRunningTime="2026-09-09 12:32:36.87801691 +0000 UTC m=+1279.508604742"
Sep 09 12:32:42.320521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:42.320479    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:42.321002 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:42.320539    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:42.333310 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:42.333283    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:42.875907 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:42.875866    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:32:47.840947 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:47.840911    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:32:47.841347 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:32:47.841089    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:34:43.103197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.103162    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:34:43.103820 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.103387    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" podUID="455b70d5-b242-4684-b988-dafa469b1e0c" containerName="payload-processing" containerID="cri-o://559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859" gracePeriod=30
Sep 09 12:34:43.116911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.116881    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:34:43.117173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.117124    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" podUID="5b6f1fb7-7094-4bab-b115-87a6aa463596" containerName="payload-pre-processing" containerID="cri-o://19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870" gracePeriod=30
Sep 09 12:34:43.387870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.387851    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:34:43.401993 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.401968    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:34:43.506695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.506656    2556 generic.go:356] "Generic (PLEG): container finished" podID="5b6f1fb7-7094-4bab-b115-87a6aa463596" containerID="19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870" exitCode=0
Sep 09 12:34:43.506875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.506709    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"
Sep 09 12:34:43.506875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.506741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" event={"ID":"5b6f1fb7-7094-4bab-b115-87a6aa463596","Type":"ContainerDied","Data":"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"}
Sep 09 12:34:43.506875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.506793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4" event={"ID":"5b6f1fb7-7094-4bab-b115-87a6aa463596","Type":"ContainerDied","Data":"f225626e39b063c8949a70bf548501e5daef7b70eedfe40b2c8ed2d6c8cc10c8"}
Sep 09 12:34:43.506875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.506810    2556 scope.go:117] "RemoveContainer" containerID="19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"
Sep 09 12:34:43.508125 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.508093    2556 generic.go:356] "Generic (PLEG): container finished" podID="455b70d5-b242-4684-b988-dafa469b1e0c" containerID="559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859" exitCode=0
Sep 09 12:34:43.508202 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.508135    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"
Sep 09 12:34:43.508202 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.508167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" event={"ID":"455b70d5-b242-4684-b988-dafa469b1e0c","Type":"ContainerDied","Data":"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"}
Sep 09 12:34:43.508319 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.508206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp" event={"ID":"455b70d5-b242-4684-b988-dafa469b1e0c","Type":"ContainerDied","Data":"e815364a4aa95bf1c22b65f914ca49d2c298a5309a317a0ac5a33c94e0f44315"}
Sep 09 12:34:43.516207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.516189    2556 scope.go:117] "RemoveContainer" containerID="19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"
Sep 09 12:34:43.516531 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:34:43.516504    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870\": container with ID starting with 19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870 not found: ID does not exist" containerID="19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"
Sep 09 12:34:43.516579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.516534    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870"} err="failed to get container status \"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870\": rpc error: code = NotFound desc = could not find container \"19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870\": container with ID starting with 19f58bad7ce698ff8c21b42af38c7d679a5db96bcef134d09c581ad28b555870 not found: ID does not exist"
Sep 09 12:34:43.516579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.516551    2556 scope.go:117] "RemoveContainer" containerID="559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"
Sep 09 12:34:43.524882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.524864    2556 scope.go:117] "RemoveContainer" containerID="559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"
Sep 09 12:34:43.525163 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:34:43.525137    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859\": container with ID starting with 559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859 not found: ID does not exist" containerID="559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"
Sep 09 12:34:43.525232 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.525162    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859"} err="failed to get container status \"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859\": rpc error: code = NotFound desc = could not find container \"559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859\": container with ID starting with 559a63a9ee9f64f9fd0c072d40389ec78eb7da8e8e1aa2d07fc01cc3f193b859 not found: ID does not exist"
Sep 09 12:34:43.561551 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.561519    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jwc5\" (UniqueName: \"kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5\") pod \"455b70d5-b242-4684-b988-dafa469b1e0c\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") "
Sep 09 12:34:43.561686 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.561575    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume\") pod \"5b6f1fb7-7094-4bab-b115-87a6aa463596\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") "
Sep 09 12:34:43.561686 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.561674    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume\") pod \"455b70d5-b242-4684-b988-dafa469b1e0c\" (UID: \"455b70d5-b242-4684-b988-dafa469b1e0c\") "
Sep 09 12:34:43.561758 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.561696    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shzsp\" (UniqueName: \"kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp\") pod \"5b6f1fb7-7094-4bab-b115-87a6aa463596\" (UID: \"5b6f1fb7-7094-4bab-b115-87a6aa463596\") "
Sep 09 12:34:43.562048 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.562009    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b6f1fb7-7094-4bab-b115-87a6aa463596" (UID: "5b6f1fb7-7094-4bab-b115-87a6aa463596"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:43.562169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.562047    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "455b70d5-b242-4684-b988-dafa469b1e0c" (UID: "455b70d5-b242-4684-b988-dafa469b1e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:43.563860 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.563839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5" (OuterVolumeSpecName: "kube-api-access-8jwc5") pod "455b70d5-b242-4684-b988-dafa469b1e0c" (UID: "455b70d5-b242-4684-b988-dafa469b1e0c"). InnerVolumeSpecName "kube-api-access-8jwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:43.563991 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.563971    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp" (OuterVolumeSpecName: "kube-api-access-shzsp") pod "5b6f1fb7-7094-4bab-b115-87a6aa463596" (UID: "5b6f1fb7-7094-4bab-b115-87a6aa463596"). InnerVolumeSpecName "kube-api-access-shzsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:34:43.663349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.663221    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwc5\" (UniqueName: \"kubernetes.io/projected/455b70d5-b242-4684-b988-dafa469b1e0c-kube-api-access-8jwc5\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:34:43.663349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.663283    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6f1fb7-7094-4bab-b115-87a6aa463596-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:34:43.663349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.663298    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455b70d5-b242-4684-b988-dafa469b1e0c-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:34:43.663349 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.663311    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzsp\" (UniqueName: \"kubernetes.io/projected/5b6f1fb7-7094-4bab-b115-87a6aa463596-kube-api-access-shzsp\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:34:43.832881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.832845    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:34:43.837674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.837647    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24ec44a5-74dff89646-nf4f4"]
Sep 09 12:34:43.849947 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.849920    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:34:43.854901 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.854878    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24ec44a5-b999c76b5-x48bp"]
Sep 09 12:34:43.902646 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.902617    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455b70d5-b242-4684-b988-dafa469b1e0c" path="/var/lib/kubelet/pods/455b70d5-b242-4684-b988-dafa469b1e0c/volumes"
Sep 09 12:34:43.902999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:43.902986    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6f1fb7-7094-4bab-b115-87a6aa463596" path="/var/lib/kubelet/pods/5b6f1fb7-7094-4bab-b115-87a6aa463596/volumes"
Sep 09 12:34:52.113747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:52.113713    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:34:57.138257 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138216    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"]
Sep 09 12:34:57.138640 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138599    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455b70d5-b242-4684-b988-dafa469b1e0c" containerName="payload-processing"
Sep 09 12:34:57.138640 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138610    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="455b70d5-b242-4684-b988-dafa469b1e0c" containerName="payload-processing"
Sep 09 12:34:57.138640 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138624    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6f1fb7-7094-4bab-b115-87a6aa463596" containerName="payload-pre-processing"
Sep 09 12:34:57.138640 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138630    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6f1fb7-7094-4bab-b115-87a6aa463596" containerName="payload-pre-processing"
Sep 09 12:34:57.138771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138700    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6f1fb7-7094-4bab-b115-87a6aa463596" containerName="payload-pre-processing"
Sep 09 12:34:57.138771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.138708    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="455b70d5-b242-4684-b988-dafa469b1e0c" containerName="payload-processing"
Sep 09 12:34:57.141691 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.141673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:57.143870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.143847    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:34:57.143983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.143892    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:34:57.149661 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.149640    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"]
Sep 09 12:34:57.190742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.190702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk\" (UID: \"23978e2b-714d-46ae-9d78-1092afc603df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:57.291551 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.291492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk\" (UID: \"23978e2b-714d-46ae-9d78-1092afc603df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:57.292138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.292117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk\" (UID: \"23978e2b-714d-46ae-9d78-1092afc603df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:57.453519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.453432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:57.727698 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:57.727667    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"]
Sep 09 12:34:57.730541 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:34:57.730509    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23978e2b_714d_46ae_9d78_1092afc603df.slice/crio-612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf WatchSource:0}: Error finding container 612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf: Status 404 returned error can't find the container with id 612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf
Sep 09 12:34:58.596984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:58.596947    2556 generic.go:356] "Generic (PLEG): container finished" podID="23978e2b-714d-46ae-9d78-1092afc603df" containerID="150381c406e0c75f17465805f6b3d7012c647843b4c049b6c64123b71cd767c3" exitCode=0
Sep 09 12:34:58.597365 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:58.596998    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk" event={"ID":"23978e2b-714d-46ae-9d78-1092afc603df","Type":"ContainerDied","Data":"150381c406e0c75f17465805f6b3d7012c647843b4c049b6c64123b71cd767c3"}
Sep 09 12:34:58.597365 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:58.597055    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk" event={"ID":"23978e2b-714d-46ae-9d78-1092afc603df","Type":"ContainerStarted","Data":"612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf"}
Sep 09 12:34:59.739066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:59.739038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:34:59.817701 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:59.817660    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca\") pod \"23978e2b-714d-46ae-9d78-1092afc603df\" (UID: \"23978e2b-714d-46ae-9d78-1092afc603df\") "
Sep 09 12:34:59.818072 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:59.818047    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23978e2b-714d-46ae-9d78-1092afc603df" (UID: "23978e2b-714d-46ae-9d78-1092afc603df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:34:59.919175 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:34:59.919144    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23978e2b-714d-46ae-9d78-1092afc603df-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:00.609695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:00.609652    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk" event={"ID":"23978e2b-714d-46ae-9d78-1092afc603df","Type":"ContainerDied","Data":"612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf"}
Sep 09 12:35:00.609695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:00.609683    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"
Sep 09 12:35:00.609909 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:00.609715    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="612095750e813a2d1cc9ec230ac851f07da29fe6c5c02d160f949a65aad08bbf"
Sep 09 12:35:02.011433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:02.011397    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:35:03.491991 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.491937    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:35:03.492904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.492878    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23978e2b-714d-46ae-9d78-1092afc603df" containerName="revoke-keys"
Sep 09 12:35:03.492904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.492904    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="23978e2b-714d-46ae-9d78-1092afc603df" containerName="revoke-keys"
Sep 09 12:35:03.493084 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.493070    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="23978e2b-714d-46ae-9d78-1092afc603df" containerName="revoke-keys"
Sep 09 12:35:03.497338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.497303    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.500971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.500940    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"default-dockercfg-cp998\""
Sep 09 12:35:03.501120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.500969    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"kube-root-ca.crt\""
Sep 09 12:35:03.501120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.500991    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"auth-test-model-cfcdfd-kserve-self-signed-certs\""
Sep 09 12:35:03.501120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.501063    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"openshift-service-ca.crt\""
Sep 09 12:35:03.505743 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.505721    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:35:03.550323 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.550323 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.550586 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550353    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.550586 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.550586 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550506    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.550586 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.550533    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpcx\" (UniqueName: \"kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651258 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651305    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651352    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpcx\" (UniqueName: \"kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651664    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651739 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.651887 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.651866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.653699 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.653677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.654065 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.654046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.661775 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.661748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpcx\" (UniqueName: \"kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx\") pod \"auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:03.808310 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:03.808203    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:04.085655 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:04.085632    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:35:04.087574 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:35:04.087547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d773642_185d_4ee1_a621_2036f6a2e336.slice/crio-9cf865f8dfe672d6d3a1309e797c1ba44b084349e1ea7a54730638d536a06838 WatchSource:0}: Error finding container 9cf865f8dfe672d6d3a1309e797c1ba44b084349e1ea7a54730638d536a06838: Status 404 returned error can't find the container with id 9cf865f8dfe672d6d3a1309e797c1ba44b084349e1ea7a54730638d536a06838
Sep 09 12:35:04.631980 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:04.631945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" event={"ID":"0d773642-185d-4ee1-a621-2036f6a2e336","Type":"ContainerStarted","Data":"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"}
Sep 09 12:35:04.632383 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:04.631992    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" event={"ID":"0d773642-185d-4ee1-a621-2036f6a2e336","Type":"ContainerStarted","Data":"9cf865f8dfe672d6d3a1309e797c1ba44b084349e1ea7a54730638d536a06838"}
Sep 09 12:35:04.651623 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:04.651573    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" podStartSLOduration=1.651555701 podStartE2EDuration="1.651555701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:35:04.650518831 +0000 UTC m=+1427.281106608" watchObservedRunningTime="2026-09-09 12:35:04.651555701 +0000 UTC m=+1427.282143477"
Sep 09 12:35:07.134478 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.134408    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" podUID="9c43429c-e9a7-4d4b-9429-6703e9a37669" containerName="main" containerID="cri-o://f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663" gracePeriod=45
Sep 09 12:35:07.383781 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.383756    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:35:07.488980 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.488944    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p46vr\" (UniqueName: \"kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489009    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489125    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489263    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489309    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home\") pod \"9c43429c-e9a7-4d4b-9429-6703e9a37669\" (UID: \"9c43429c-e9a7-4d4b-9429-6703e9a37669\") "
Sep 09 12:35:07.489390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489331    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:07.489534 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489396    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:07.489633 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489606    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home" (OuterVolumeSpecName: "home") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:07.489633 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489618    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.489983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.489650    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.491539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.491510    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:07.491539 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.491519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr" (OuterVolumeSpecName: "kube-api-access-p46vr") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "kube-api-access-p46vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:07.491719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.491558    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm" (OuterVolumeSpecName: "dshm") pod "9c43429c-e9a7-4d4b-9429-6703e9a37669" (UID: "9c43429c-e9a7-4d4b-9429-6703e9a37669"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:07.590919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.590883    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.590919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.590911    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c43429c-e9a7-4d4b-9429-6703e9a37669-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.590919 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.590921    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p46vr\" (UniqueName: \"kubernetes.io/projected/9c43429c-e9a7-4d4b-9429-6703e9a37669-kube-api-access-p46vr\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.591164 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.590931    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c43429c-e9a7-4d4b-9429-6703e9a37669-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:07.648830 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.648797    2556 generic.go:356] "Generic (PLEG): container finished" podID="9c43429c-e9a7-4d4b-9429-6703e9a37669" containerID="f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663" exitCode=0
Sep 09 12:35:07.649026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.648854    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"
Sep 09 12:35:07.649026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.648882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" event={"ID":"9c43429c-e9a7-4d4b-9429-6703e9a37669","Type":"ContainerDied","Data":"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"}
Sep 09 12:35:07.649026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.648923    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26" event={"ID":"9c43429c-e9a7-4d4b-9429-6703e9a37669","Type":"ContainerDied","Data":"b887ad84b11544dce5a02bf1f39cf47c4e59dacbf78c7f918a4552378201dac4"}
Sep 09 12:35:07.649026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.648940    2556 scope.go:117] "RemoveContainer" containerID="f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"
Sep 09 12:35:07.658831 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.658807    2556 scope.go:117] "RemoveContainer" containerID="f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"
Sep 09 12:35:07.659130 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:35:07.659110    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663\": container with ID starting with f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663 not found: ID does not exist" containerID="f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"
Sep 09 12:35:07.659201 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.659142    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663"} err="failed to get container status \"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663\": rpc error: code = NotFound desc = could not find container \"f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663\": container with ID starting with f23b9edb4d9d41144ea364569fe950c2c0da8d043e7814e6f37b1c4d1a8d9663 not found: ID does not exist"
Sep 09 12:35:07.670324 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.670297    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:35:07.676940 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.676916    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-24ec44a5/e2e-lifecycle-model-24ec44a5-kserve-7cf99bcb75-l4d26"]
Sep 09 12:35:07.909420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:07.909334    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c43429c-e9a7-4d4b-9429-6703e9a37669" path="/var/lib/kubelet/pods/9c43429c-e9a7-4d4b-9429-6703e9a37669/volumes"
Sep 09 12:35:13.299299 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.299180    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"]
Sep 09 12:35:13.301287 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.301249    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c554db-29815950-q6txs"]
Sep 09 12:35:13.326131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.326098    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:35:13.326372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.326320    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" podUID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" containerName="payload-processing" containerID="cri-o://a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47" gracePeriod=30
Sep 09 12:35:13.519578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.519542    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"]
Sep 09 12:35:13.526719 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.526685    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c554db-zmtwk"]
Sep 09 12:35:13.584630 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.584603    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:35:13.648518 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.648486    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d72s\" (UniqueName: \"kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s\") pod \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") "
Sep 09 12:35:13.648666 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.648617    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume\") pod \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\" (UID: \"3c75f6c6-5e92-47f8-8b38-9a2a3b161770\") "
Sep 09 12:35:13.648972 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.648951    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c75f6c6-5e92-47f8-8b38-9a2a3b161770" (UID: "3c75f6c6-5e92-47f8-8b38-9a2a3b161770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:35:13.650758 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.650734    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s" (OuterVolumeSpecName: "kube-api-access-5d72s") pod "3c75f6c6-5e92-47f8-8b38-9a2a3b161770" (UID: "3c75f6c6-5e92-47f8-8b38-9a2a3b161770"). InnerVolumeSpecName "kube-api-access-5d72s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:13.690089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.690054    2556 generic.go:356] "Generic (PLEG): container finished" podID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" containerID="a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47" exitCode=0
Sep 09 12:35:13.690300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.690122    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"
Sep 09 12:35:13.690300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.690149    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" event={"ID":"3c75f6c6-5e92-47f8-8b38-9a2a3b161770","Type":"ContainerDied","Data":"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"}
Sep 09 12:35:13.690300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.690252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp" event={"ID":"3c75f6c6-5e92-47f8-8b38-9a2a3b161770","Type":"ContainerDied","Data":"77aad947a66a0b019b2997011d64b7ca5f2bce849660846d18d324d911c49fe9"}
Sep 09 12:35:13.690300 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.690279    2556 scope.go:117] "RemoveContainer" containerID="a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"
Sep 09 12:35:13.699966 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.699944    2556 scope.go:117] "RemoveContainer" containerID="a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"
Sep 09 12:35:13.700290 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:35:13.700261    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47\": container with ID starting with a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47 not found: ID does not exist" containerID="a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"
Sep 09 12:35:13.700393 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.700291    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47"} err="failed to get container status \"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47\": rpc error: code = NotFound desc = could not find container \"a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47\": container with ID starting with a643306b3a117eb71822316e68b9e0340f89198e202b35e84c728f692cdd6f47 not found: ID does not exist"
Sep 09 12:35:13.712674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.712647    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:35:13.716840 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.716819    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c554db-58d94969fc-vkhrp"]
Sep 09 12:35:13.750024 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.749993    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d72s\" (UniqueName: \"kubernetes.io/projected/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-kube-api-access-5d72s\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:13.750024 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.750021    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c75f6c6-5e92-47f8-8b38-9a2a3b161770-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:13.808634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.808595    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:13.808634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.808644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:13.821627 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.821549    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:13.901881 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.901845    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed" path="/var/lib/kubelet/pods/0a53b9de-7f7c-49eb-aba9-85fce2d4b9ed/volumes"
Sep 09 12:35:13.902165 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.902152    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23978e2b-714d-46ae-9d78-1092afc603df" path="/var/lib/kubelet/pods/23978e2b-714d-46ae-9d78-1092afc603df/volumes"
Sep 09 12:35:13.902524 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:13.902511    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" path="/var/lib/kubelet/pods/3c75f6c6-5e92-47f8-8b38-9a2a3b161770/volumes"
Sep 09 12:35:14.708440 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:14.708412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:35:17.032199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.032151    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" podUID="cd99e616-1fa6-42aa-8568-e1460a3361a9" containerName="main" containerID="cri-o://c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5" gracePeriod=45
Sep 09 12:35:17.287123 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.287061    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:35:17.387441 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387411    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387528    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387564    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnrbn\" (UniqueName: \"kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387629 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387591    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387829 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387829 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387718    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache\") pod \"cd99e616-1fa6-42aa-8568-e1460a3361a9\" (UID: \"cd99e616-1fa6-42aa-8568-e1460a3361a9\") "
Sep 09 12:35:17.387829 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387737    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:17.387979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.387831    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home" (OuterVolumeSpecName: "home") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:17.388133 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.388104    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.388305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.388139    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.388305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.388098    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:17.390624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.390592    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn" (OuterVolumeSpecName: "kube-api-access-wnrbn") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "kube-api-access-wnrbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:35:17.390742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.390702    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:35:17.390742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.390713    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm" (OuterVolumeSpecName: "dshm") pod "cd99e616-1fa6-42aa-8568-e1460a3361a9" (UID: "cd99e616-1fa6-42aa-8568-e1460a3361a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:35:17.489029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.488986    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd99e616-1fa6-42aa-8568-e1460a3361a9-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.489029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.489022    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.489029 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.489035    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnrbn\" (UniqueName: \"kubernetes.io/projected/cd99e616-1fa6-42aa-8568-e1460a3361a9-kube-api-access-wnrbn\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.489302 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.489048    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd99e616-1fa6-42aa-8568-e1460a3361a9-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:35:17.713536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.713501    2556 generic.go:356] "Generic (PLEG): container finished" podID="cd99e616-1fa6-42aa-8568-e1460a3361a9" containerID="c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5" exitCode=0
Sep 09 12:35:17.713718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.713542    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" event={"ID":"cd99e616-1fa6-42aa-8568-e1460a3361a9","Type":"ContainerDied","Data":"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"}
Sep 09 12:35:17.713718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.713552    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"
Sep 09 12:35:17.713718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.713570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x" event={"ID":"cd99e616-1fa6-42aa-8568-e1460a3361a9","Type":"ContainerDied","Data":"109b5f2882fbb897d8bfd1e0c40a00d02f316ab056e87f83edaf6563d954a093"}
Sep 09 12:35:17.713718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.713587    2556 scope.go:117] "RemoveContainer" containerID="c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"
Sep 09 12:35:17.723889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.723865    2556 scope.go:117] "RemoveContainer" containerID="c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"
Sep 09 12:35:17.724209 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:35:17.724186    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5\": container with ID starting with c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5 not found: ID does not exist" containerID="c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"
Sep 09 12:35:17.724285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.724224    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5"} err="failed to get container status \"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5\": rpc error: code = NotFound desc = could not find container \"c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5\": container with ID starting with c8b30a475a9c093fcaed2833c3c3c80d6654855ab8b19a3e7e8c37464f9657c5 not found: ID does not exist"
Sep 09 12:35:17.735273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.735227    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:35:17.739578 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.739555    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c554db/simulator-w0-c554db-kserve-7cc87b4cfb-wgl9x"]
Sep 09 12:35:17.906690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:35:17.906659    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd99e616-1fa6-42aa-8568-e1460a3361a9" path="/var/lib/kubelet/pods/cd99e616-1fa6-42aa-8568-e1460a3361a9/volumes"
Sep 09 12:36:13.807889 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.807855    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808270    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c43429c-e9a7-4d4b-9429-6703e9a37669" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808282    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c43429c-e9a7-4d4b-9429-6703e9a37669" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808300    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" containerName="payload-processing"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808309    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" containerName="payload-processing"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808319    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd99e616-1fa6-42aa-8568-e1460a3361a9" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808325    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd99e616-1fa6-42aa-8568-e1460a3361a9" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808405    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c43429c-e9a7-4d4b-9429-6703e9a37669" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808415    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd99e616-1fa6-42aa-8568-e1460a3361a9" containerName="main"
Sep 09 12:36:13.808472 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.808423    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c75f6c6-5e92-47f8-8b38-9a2a3b161770" containerName="payload-processing"
Sep 09 12:36:13.811507 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.811486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.814477 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.814456    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc0270\"/\"e2e-conflict-b-67a70fd9-kserve-self-signed-certs\""
Sep 09 12:36:13.823766 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.823742    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:13.918979 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.918942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.919172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.918988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.919172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.919004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.919172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.919059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqdc\" (UniqueName: \"kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.919172 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.919080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:13.919411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:13.919213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020727 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020737    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020833    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqdc\" (UniqueName: \"kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.020926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.020860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.021218 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.021149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.021218 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.021172    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.021354 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.021269    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.023280 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.023256    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.023467 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.023449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.029741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.029721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqdc\" (UniqueName: \"kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc\") pod \"e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") " pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.123293 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.123180    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:14.397197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:14.397150    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:14.399800 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:36:14.399771    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e933192_837a_4745_ba63_a3131fefd114.slice/crio-5a7c393fb0d7979bcc8f4f33b5c151461160216973b4d3f0756402ce900e2b18 WatchSource:0}: Error finding container 5a7c393fb0d7979bcc8f4f33b5c151461160216973b4d3f0756402ce900e2b18: Status 404 returned error can't find the container with id 5a7c393fb0d7979bcc8f4f33b5c151461160216973b4d3f0756402ce900e2b18
Sep 09 12:36:15.010779 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:15.010740    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" event={"ID":"9e933192-837a-4745-ba63-a3131fefd114","Type":"ContainerStarted","Data":"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"}
Sep 09 12:36:15.010779 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:15.010782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" event={"ID":"9e933192-837a-4745-ba63-a3131fefd114","Type":"ContainerStarted","Data":"5a7c393fb0d7979bcc8f4f33b5c151461160216973b4d3f0756402ce900e2b18"}
Sep 09 12:36:15.035773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:15.033481    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" podStartSLOduration=2.033455257 podStartE2EDuration="2.033455257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:15.029438701 +0000 UTC m=+1497.660026488" watchObservedRunningTime="2026-09-09 12:36:15.033455257 +0000 UTC m=+1497.664043034"
Sep 09 12:36:17.885621 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:17.885590    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:36:17.887950 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:17.887915    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:36:17.890099 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:17.890079    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:36:17.892113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:17.892092    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:36:18.166307 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:18.166210    2556 scope.go:117] "RemoveContainer" containerID="77a4e4da824a5926c49fac37c16c99889df194cb93ae441be9ab6c8ce39e8787"
Sep 09 12:36:18.174663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:18.174642    2556 scope.go:117] "RemoveContainer" containerID="aa413827d274d75305d95cad479bdcc9797107d600d707d5dc2adbf56daf740b"
Sep 09 12:36:18.182592 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:18.182572    2556 scope.go:117] "RemoveContainer" containerID="5135db83c6ae93985723f7376a8f6ce64aa188a768e7953fee552d595ca91b3b"
Sep 09 12:36:24.123303 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:24.123273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:24.123904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:24.123322    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:24.136760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:24.136729    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:25.071613 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:25.071585    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:37.490273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:37.490216    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:47.238924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.238839    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:36:47.247207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.247174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.250565 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.250534    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:36:47.250760 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.250741    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-364341\""
Sep 09 12:36:47.250847 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.250775    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-364341-dockercfg-r862w\""
Sep 09 12:36:47.335125 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.335079    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.335334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.335232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjk7\" (UniqueName: \"kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.436635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.436595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjk7\" (UniqueName: \"kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.436846 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.436664    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.437348 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.437331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.446781 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.446754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjk7\" (UniqueName: \"kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7\") pod \"payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.560129 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.560039    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:47.832599 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:47.832506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:36:47.835759 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:36:47.835726    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5defbf6_c1fc_471a_b937_b5bdc2cd6b6e.slice/crio-0ae6ac64a1b3c8e4b33c970b7fe1d5e27be7102f97dfc2699bd3f9a81bc827e6 WatchSource:0}: Error finding container 0ae6ac64a1b3c8e4b33c970b7fe1d5e27be7102f97dfc2699bd3f9a81bc827e6: Status 404 returned error can't find the container with id 0ae6ac64a1b3c8e4b33c970b7fe1d5e27be7102f97dfc2699bd3f9a81bc827e6
Sep 09 12:36:48.175616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:48.175520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" event={"ID":"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e","Type":"ContainerStarted","Data":"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"}
Sep 09 12:36:48.175616 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:48.175582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" event={"ID":"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e","Type":"ContainerStarted","Data":"0ae6ac64a1b3c8e4b33c970b7fe1d5e27be7102f97dfc2699bd3f9a81bc827e6"}
Sep 09 12:36:48.175806 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:48.175659    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:36:52.511024 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.510960    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" podUID="9e933192-837a-4745-ba63-a3131fefd114" containerName="main" containerID="cri-o://6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67" gracePeriod=45
Sep 09 12:36:52.770301 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.770199    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:52.787880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.787823    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" podStartSLOduration=5.787804064 podStartE2EDuration="5.787804064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:36:48.191343006 +0000 UTC m=+1530.821930783" watchObservedRunningTime="2026-09-09 12:36:52.787804064 +0000 UTC m=+1535.418391841"
Sep 09 12:36:52.792372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792458    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtqdc\" (UniqueName: \"kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792526    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792565    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792613    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792636    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm\") pod \"9e933192-837a-4745-ba63-a3131fefd114\" (UID: \"9e933192-837a-4745-ba63-a3131fefd114\") "
Sep 09 12:36:52.792724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792644    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:52.792896 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.792876    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:52.793182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.793086    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home" (OuterVolumeSpecName: "home") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:52.793508 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.793223    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:52.794968 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.794941    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:36:52.795088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.794977    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc" (OuterVolumeSpecName: "kube-api-access-jtqdc") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "kube-api-access-jtqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:36:52.795088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.795023    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm" (OuterVolumeSpecName: "dshm") pod "9e933192-837a-4745-ba63-a3131fefd114" (UID: "9e933192-837a-4745-ba63-a3131fefd114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:36:52.893677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.893641    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtqdc\" (UniqueName: \"kubernetes.io/projected/9e933192-837a-4745-ba63-a3131fefd114-kube-api-access-jtqdc\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:52.893677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.893671    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:52.893677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.893682    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e933192-837a-4745-ba63-a3131fefd114-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:52.893924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.893691    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:52.893924 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:52.893699    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e933192-837a-4745-ba63-a3131fefd114-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:36:53.202288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.202184    2556 generic.go:356] "Generic (PLEG): container finished" podID="9e933192-837a-4745-ba63-a3131fefd114" containerID="6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67" exitCode=0
Sep 09 12:36:53.202288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.202273    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"
Sep 09 12:36:53.202288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.202276    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" event={"ID":"9e933192-837a-4745-ba63-a3131fefd114","Type":"ContainerDied","Data":"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"}
Sep 09 12:36:53.202571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.202328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq" event={"ID":"9e933192-837a-4745-ba63-a3131fefd114","Type":"ContainerDied","Data":"5a7c393fb0d7979bcc8f4f33b5c151461160216973b4d3f0756402ce900e2b18"}
Sep 09 12:36:53.202571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.202350    2556 scope.go:117] "RemoveContainer" containerID="6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"
Sep 09 12:36:53.212454 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.212435    2556 scope.go:117] "RemoveContainer" containerID="6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"
Sep 09 12:36:53.212782 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:36:53.212765    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67\": container with ID starting with 6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67 not found: ID does not exist" containerID="6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"
Sep 09 12:36:53.212834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.212792    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67"} err="failed to get container status \"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67\": rpc error: code = NotFound desc = could not find container \"6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67\": container with ID starting with 6b48d60ed71460a262fa9ecf66f24ed94b64f22a1d39235eabf30545ebd01d67 not found: ID does not exist"
Sep 09 12:36:53.227584 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.227544    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:53.231918 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.231889    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/e2e-conflict-b-67a70fd9-kserve-58dddf5c49-wgfzq"]
Sep 09 12:36:53.902717 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:53.902685    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e933192-837a-4745-ba63-a3131fefd114" path="/var/lib/kubelet/pods/9e933192-837a-4745-ba63-a3131fefd114/volumes"
Sep 09 12:36:59.182970 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:36:59.182931    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:37:16.170278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.170230    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:37:16.172700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.170599    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e933192-837a-4745-ba63-a3131fefd114" containerName="main"
Sep 09 12:37:16.172700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.170610    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e933192-837a-4745-ba63-a3131fefd114" containerName="main"
Sep 09 12:37:16.172700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.170683    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e933192-837a-4745-ba63-a3131fefd114" containerName="main"
Sep 09 12:37:16.173654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.173635    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.176484 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.176464    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-59ce1d-dockercfg-xvl5b\""
Sep 09 12:37:16.176618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.176484    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-59ce1d\""
Sep 09 12:37:16.183723 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.183686    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:37:16.197742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.197710    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:37:16.207610 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.207576    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.210064 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.210031    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:37:16.308036 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.307992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.308228 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.308061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.308228 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.308088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4mj\" (UniqueName: \"kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.308228 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.308182    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4q4w\" (UniqueName: \"kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.408995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.408947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4q4w\" (UniqueName: \"kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.409207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.409057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.409207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.409107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.409207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.409134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4mj\" (UniqueName: \"kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.409870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.409848    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.409870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.409861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.418895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.418865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4mj\" (UniqueName: \"kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj\") pod \"payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.419009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.418865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4q4w\" (UniqueName: \"kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w\") pod \"payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") " pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.485411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.485370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:16.519439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.519401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:16.759010 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.758983    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:37:16.761132 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:37:16.761097    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21eadcf_fc29_4116_9847_7ce11d3fcc6e.slice/crio-7a06f0b74e20dae93da2fc2b11138f754e2dd2b3e60b41de41c2d944176301be WatchSource:0}: Error finding container 7a06f0b74e20dae93da2fc2b11138f754e2dd2b3e60b41de41c2d944176301be: Status 404 returned error can't find the container with id 7a06f0b74e20dae93da2fc2b11138f754e2dd2b3e60b41de41c2d944176301be
Sep 09 12:37:16.802725 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:16.802698    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:37:16.808435 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:37:16.808406    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab79640_4126_45f8_8e05_7985a98cad10.slice/crio-cdad2b4ca5dbed69127997d542d99c095a04028597d7ad88c52436c5fa7afe51 WatchSource:0}: Error finding container cdad2b4ca5dbed69127997d542d99c095a04028597d7ad88c52436c5fa7afe51: Status 404 returned error can't find the container with id cdad2b4ca5dbed69127997d542d99c095a04028597d7ad88c52436c5fa7afe51
Sep 09 12:37:17.327543 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.327502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" event={"ID":"b21eadcf-fc29-4116-9847-7ce11d3fcc6e","Type":"ContainerStarted","Data":"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"}
Sep 09 12:37:17.328058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.327575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" event={"ID":"b21eadcf-fc29-4116-9847-7ce11d3fcc6e","Type":"ContainerStarted","Data":"7a06f0b74e20dae93da2fc2b11138f754e2dd2b3e60b41de41c2d944176301be"}
Sep 09 12:37:17.328058 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.327604    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:17.329430 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.329400    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" event={"ID":"4ab79640-4126-45f8-8e05-7985a98cad10","Type":"ContainerStarted","Data":"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"}
Sep 09 12:37:17.329563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.329458    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:17.329563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.329482    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" event={"ID":"4ab79640-4126-45f8-8e05-7985a98cad10","Type":"ContainerStarted","Data":"cdad2b4ca5dbed69127997d542d99c095a04028597d7ad88c52436c5fa7afe51"}
Sep 09 12:37:17.345256 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.345193    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" podStartSLOduration=1.345177912 podStartE2EDuration="1.345177912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:37:17.343961307 +0000 UTC m=+1559.974549084" watchObservedRunningTime="2026-09-09 12:37:17.345177912 +0000 UTC m=+1559.975765685"
Sep 09 12:37:17.364253 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:17.364178    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" podStartSLOduration=1.364156961 podStartE2EDuration="1.364156961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:37:17.361660179 +0000 UTC m=+1559.992247960" watchObservedRunningTime="2026-09-09 12:37:17.364156961 +0000 UTC m=+1559.994744740"
Sep 09 12:37:18.208466 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:18.208435    2556 scope.go:117] "RemoveContainer" containerID="49ce5dac57b1eec2b03e49b131f43122ee248335b78ae0b57b55f03e5b8c8dda"
Sep 09 12:37:28.336347 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:28.336306    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:37:28.336755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:28.336386    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:37:58.436818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.436776    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:37:58.437588 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.437553    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" podUID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" containerName="payload-pre-processing" containerID="cri-o://7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1" gracePeriod=30
Sep 09 12:37:58.693226 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.693165    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:37:58.812137 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.812103    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume\") pod \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") "
Sep 09 12:37:58.812334 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.812159    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sjk7\" (UniqueName: \"kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7\") pod \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\" (UID: \"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e\") "
Sep 09 12:37:58.812600 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.812565    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" (UID: "a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:37:58.814422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.814397    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7" (OuterVolumeSpecName: "kube-api-access-9sjk7") pod "a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" (UID: "a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e"). InnerVolumeSpecName "kube-api-access-9sjk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:37:58.913168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.913127    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:37:58.913168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:58.913165    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjk7\" (UniqueName: \"kubernetes.io/projected/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e-kube-api-access-9sjk7\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:37:59.549253 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.549205    2556 generic.go:356] "Generic (PLEG): container finished" podID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" containerID="7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1" exitCode=0
Sep 09 12:37:59.549658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.549273    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"
Sep 09 12:37:59.549658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.549277    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" event={"ID":"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e","Type":"ContainerDied","Data":"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"}
Sep 09 12:37:59.549658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.549335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb" event={"ID":"a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e","Type":"ContainerDied","Data":"0ae6ac64a1b3c8e4b33c970b7fe1d5e27be7102f97dfc2699bd3f9a81bc827e6"}
Sep 09 12:37:59.549658 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.549365    2556 scope.go:117] "RemoveContainer" containerID="7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"
Sep 09 12:37:59.558231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.558216    2556 scope.go:117] "RemoveContainer" containerID="7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"
Sep 09 12:37:59.558528 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:37:59.558509    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1\": container with ID starting with 7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1 not found: ID does not exist" containerID="7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"
Sep 09 12:37:59.558573 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.558536    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1"} err="failed to get container status \"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1\": rpc error: code = NotFound desc = could not find container \"7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1\": container with ID starting with 7a287ed30cda8c8222b8c533889cdbe5ba6c0c5a564b73b14eaa2c3acef7e9a1 not found: ID does not exist"
Sep 09 12:37:59.571391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.571367    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:37:59.580433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.580412    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-364341-8694679854-rkbzb"]
Sep 09 12:37:59.902699 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:37:59.902623    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" path="/var/lib/kubelet/pods/a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e/volumes"
Sep 09 12:38:21.624876 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.624841    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:38:21.625368 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.625070    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" podUID="4ab79640-4126-45f8-8e05-7985a98cad10" containerName="payload-processing" containerID="cri-o://865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459" gracePeriod=30
Sep 09 12:38:21.631548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.631516    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:38:21.631778 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.631751    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" podUID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" containerName="payload-pre-processing" containerID="cri-o://b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d" gracePeriod=30
Sep 09 12:38:21.912675 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.912649    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:38:21.915461 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.915442    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:38:21.928994 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.928971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume\") pod \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") "
Sep 09 12:38:21.929074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.929006    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4q4w\" (UniqueName: \"kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w\") pod \"4ab79640-4126-45f8-8e05-7985a98cad10\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") "
Sep 09 12:38:21.929074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.929033    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume\") pod \"4ab79640-4126-45f8-8e05-7985a98cad10\" (UID: \"4ab79640-4126-45f8-8e05-7985a98cad10\") "
Sep 09 12:38:21.929074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.929072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw4mj\" (UniqueName: \"kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj\") pod \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\" (UID: \"b21eadcf-fc29-4116-9847-7ce11d3fcc6e\") "
Sep 09 12:38:21.929409 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.929379    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b21eadcf-fc29-4116-9847-7ce11d3fcc6e" (UID: "b21eadcf-fc29-4116-9847-7ce11d3fcc6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:21.929513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.929429    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ab79640-4126-45f8-8e05-7985a98cad10" (UID: "4ab79640-4126-45f8-8e05-7985a98cad10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:21.931793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.931755    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj" (OuterVolumeSpecName: "kube-api-access-kw4mj") pod "b21eadcf-fc29-4116-9847-7ce11d3fcc6e" (UID: "b21eadcf-fc29-4116-9847-7ce11d3fcc6e"). InnerVolumeSpecName "kube-api-access-kw4mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:21.931907 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:21.931876    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w" (OuterVolumeSpecName: "kube-api-access-z4q4w") pod "4ab79640-4126-45f8-8e05-7985a98cad10" (UID: "4ab79640-4126-45f8-8e05-7985a98cad10"). InnerVolumeSpecName "kube-api-access-z4q4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:22.030034 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.030002    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:22.030034 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.030028    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4q4w\" (UniqueName: \"kubernetes.io/projected/4ab79640-4126-45f8-8e05-7985a98cad10-kube-api-access-z4q4w\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:22.030034 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.030038    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab79640-4126-45f8-8e05-7985a98cad10-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:22.030296 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.030046    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4mj\" (UniqueName: \"kubernetes.io/projected/b21eadcf-fc29-4116-9847-7ce11d3fcc6e-kube-api-access-kw4mj\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:22.666171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.666135    2556 generic.go:356] "Generic (PLEG): container finished" podID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" containerID="b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d" exitCode=0
Sep 09 12:38:22.666644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.666186    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"
Sep 09 12:38:22.666644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.666226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" event={"ID":"b21eadcf-fc29-4116-9847-7ce11d3fcc6e","Type":"ContainerDied","Data":"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"}
Sep 09 12:38:22.666644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.666310    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp" event={"ID":"b21eadcf-fc29-4116-9847-7ce11d3fcc6e","Type":"ContainerDied","Data":"7a06f0b74e20dae93da2fc2b11138f754e2dd2b3e60b41de41c2d944176301be"}
Sep 09 12:38:22.666644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.666332    2556 scope.go:117] "RemoveContainer" containerID="b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"
Sep 09 12:38:22.667764 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.667741    2556 generic.go:356] "Generic (PLEG): container finished" podID="4ab79640-4126-45f8-8e05-7985a98cad10" containerID="865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459" exitCode=0
Sep 09 12:38:22.667834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.667792    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"
Sep 09 12:38:22.667900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.667790    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" event={"ID":"4ab79640-4126-45f8-8e05-7985a98cad10","Type":"ContainerDied","Data":"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"}
Sep 09 12:38:22.667959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.667941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9" event={"ID":"4ab79640-4126-45f8-8e05-7985a98cad10","Type":"ContainerDied","Data":"cdad2b4ca5dbed69127997d542d99c095a04028597d7ad88c52436c5fa7afe51"}
Sep 09 12:38:22.677261 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.677216    2556 scope.go:117] "RemoveContainer" containerID="b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"
Sep 09 12:38:22.677622 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:22.677599    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d\": container with ID starting with b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d not found: ID does not exist" containerID="b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"
Sep 09 12:38:22.677716 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.677631    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d"} err="failed to get container status \"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d\": rpc error: code = NotFound desc = could not find container \"b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d\": container with ID starting with b9722d77a3b602b7157301795cb16054e1c7e7d966b1eb64f91a7593e410873d not found: ID does not exist"
Sep 09 12:38:22.677716 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.677653    2556 scope.go:117] "RemoveContainer" containerID="865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"
Sep 09 12:38:22.686283 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.686263    2556 scope.go:117] "RemoveContainer" containerID="865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"
Sep 09 12:38:22.686561 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:22.686543    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459\": container with ID starting with 865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459 not found: ID does not exist" containerID="865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"
Sep 09 12:38:22.686609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.686569    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459"} err="failed to get container status \"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459\": rpc error: code = NotFound desc = could not find container \"865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459\": container with ID starting with 865c1b439c82bc010dfa681d647d74f868a52161d333065f23510bf2a9dd8459 not found: ID does not exist"
Sep 09 12:38:22.692742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.692723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:38:22.695826 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.695806    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59ce1d-8659fd7478-m9kcp"]
Sep 09 12:38:22.709649 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.709631    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:38:22.715730 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:22.715702    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59ce1d-754cd7ff87-96gz9"]
Sep 09 12:38:23.902445 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:23.902412    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab79640-4126-45f8-8e05-7985a98cad10" path="/var/lib/kubelet/pods/4ab79640-4126-45f8-8e05-7985a98cad10/volumes"
Sep 09 12:38:23.902796 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:23.902772    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" path="/var/lib/kubelet/pods/b21eadcf-fc29-4116-9847-7ce11d3fcc6e/volumes"
Sep 09 12:38:40.255866 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.255835    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:38:40.256451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.256064    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" podUID="6ad98c68-276b-4aee-8282-dbb70fca298c" containerName="payload-processing" containerID="cri-o://5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b" gracePeriod=30
Sep 09 12:38:40.270536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.270506    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:38:40.270814 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.270744    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" podUID="45d40b59-5e24-46ef-85dd-7f85bd902430" containerName="payload-pre-processing" containerID="cri-o://3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c" gracePeriod=30
Sep 09 12:38:40.548729 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.548702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:38:40.551464 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.551444    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:38:40.607176 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume\") pod \"45d40b59-5e24-46ef-85dd-7f85bd902430\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") "
Sep 09 12:38:40.607377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607203    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume\") pod \"6ad98c68-276b-4aee-8282-dbb70fca298c\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") "
Sep 09 12:38:40.607377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v\") pod \"6ad98c68-276b-4aee-8282-dbb70fca298c\" (UID: \"6ad98c68-276b-4aee-8282-dbb70fca298c\") "
Sep 09 12:38:40.607377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607294    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw2tt\" (UniqueName: \"kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt\") pod \"45d40b59-5e24-46ef-85dd-7f85bd902430\" (UID: \"45d40b59-5e24-46ef-85dd-7f85bd902430\") "
Sep 09 12:38:40.607620 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607596    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45d40b59-5e24-46ef-85dd-7f85bd902430" (UID: "45d40b59-5e24-46ef-85dd-7f85bd902430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:40.607673 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.607614    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ad98c68-276b-4aee-8282-dbb70fca298c" (UID: "6ad98c68-276b-4aee-8282-dbb70fca298c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:40.609546 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.609520    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v" (OuterVolumeSpecName: "kube-api-access-sbs8v") pod "6ad98c68-276b-4aee-8282-dbb70fca298c" (UID: "6ad98c68-276b-4aee-8282-dbb70fca298c"). InnerVolumeSpecName "kube-api-access-sbs8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:40.609651 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.609547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt" (OuterVolumeSpecName: "kube-api-access-kw2tt") pod "45d40b59-5e24-46ef-85dd-7f85bd902430" (UID: "45d40b59-5e24-46ef-85dd-7f85bd902430"). InnerVolumeSpecName "kube-api-access-kw2tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:40.708377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.708326    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2tt\" (UniqueName: \"kubernetes.io/projected/45d40b59-5e24-46ef-85dd-7f85bd902430-kube-api-access-kw2tt\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:40.708377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.708371    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d40b59-5e24-46ef-85dd-7f85bd902430-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:40.708377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.708380    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad98c68-276b-4aee-8282-dbb70fca298c-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:40.708377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.708389    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/6ad98c68-276b-4aee-8282-dbb70fca298c-kube-api-access-sbs8v\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:40.757585 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.757549    2556 generic.go:356] "Generic (PLEG): container finished" podID="45d40b59-5e24-46ef-85dd-7f85bd902430" containerID="3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c" exitCode=0
Sep 09 12:38:40.757734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.757594    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"
Sep 09 12:38:40.757734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.757633    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" event={"ID":"45d40b59-5e24-46ef-85dd-7f85bd902430","Type":"ContainerDied","Data":"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"}
Sep 09 12:38:40.757734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.757680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv" event={"ID":"45d40b59-5e24-46ef-85dd-7f85bd902430","Type":"ContainerDied","Data":"c4ae0cbb696e8d64c5fc437c428d132394829ac03c5584069ec0beebe42da93b"}
Sep 09 12:38:40.757734 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.757714    2556 scope.go:117] "RemoveContainer" containerID="3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"
Sep 09 12:38:40.759092 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.759072    2556 generic.go:356] "Generic (PLEG): container finished" podID="6ad98c68-276b-4aee-8282-dbb70fca298c" containerID="5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b" exitCode=0
Sep 09 12:38:40.759194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.759122    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"
Sep 09 12:38:40.759194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.759126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" event={"ID":"6ad98c68-276b-4aee-8282-dbb70fca298c","Type":"ContainerDied","Data":"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"}
Sep 09 12:38:40.759194 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.759179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm" event={"ID":"6ad98c68-276b-4aee-8282-dbb70fca298c","Type":"ContainerDied","Data":"acf606943a0dc72f6e71d7fae8b6a68687b8a4c1b902a19666cc30048aa198dd"}
Sep 09 12:38:40.773991 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.773972    2556 scope.go:117] "RemoveContainer" containerID="3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"
Sep 09 12:38:40.774337 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:40.774311    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c\": container with ID starting with 3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c not found: ID does not exist" containerID="3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"
Sep 09 12:38:40.774442 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.774344    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c"} err="failed to get container status \"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c\": rpc error: code = NotFound desc = could not find container \"3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c\": container with ID starting with 3214810842fdd8282f7b425e56fd3fdd465a2e2b6f3a7dd22b66f0845d4d930c not found: ID does not exist"
Sep 09 12:38:40.774442 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.774360    2556 scope.go:117] "RemoveContainer" containerID="5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"
Sep 09 12:38:40.782278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.782258    2556 scope.go:117] "RemoveContainer" containerID="5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"
Sep 09 12:38:40.782513 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:40.782497    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b\": container with ID starting with 5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b not found: ID does not exist" containerID="5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"
Sep 09 12:38:40.782570 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.782519    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b"} err="failed to get container status \"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b\": rpc error: code = NotFound desc = could not find container \"5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b\": container with ID starting with 5297d72bc948b0de3e956508991b6e55aadaeb10ec387db72106c0cb2a6ad69b not found: ID does not exist"
Sep 09 12:38:40.785461 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.785439    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:38:40.789708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.789691    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1f639-5dc57984f4-qnvqv"]
Sep 09 12:38:40.803379 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.803356    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:38:40.809067 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:40.809047    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d1f639-55786b46c-rvrxm"]
Sep 09 12:38:41.902547 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:41.902514    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d40b59-5e24-46ef-85dd-7f85bd902430" path="/var/lib/kubelet/pods/45d40b59-5e24-46ef-85dd-7f85bd902430/volumes"
Sep 09 12:38:41.902915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:41.902875    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad98c68-276b-4aee-8282-dbb70fca298c" path="/var/lib/kubelet/pods/6ad98c68-276b-4aee-8282-dbb70fca298c/volumes"
Sep 09 12:38:43.983254 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:43.983207    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:38:58.763542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:58.763507    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:38:58.763993 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:58.763817    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" podUID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" containerName="payload-processing" containerID="cri-o://7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e" gracePeriod=30
Sep 09 12:38:58.771168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:58.770759    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:38:58.771168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:58.770990    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" podUID="ffb55623-a220-41a1-804e-d88007016ee8" containerName="payload-pre-processing" containerID="cri-o://58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3" gracePeriod=30
Sep 09 12:38:59.003388 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.003344    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" podUID="0d773642-185d-4ee1-a621-2036f6a2e336" containerName="main" containerID="cri-o://cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6" gracePeriod=45
Sep 09 12:38:59.124986 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.124961    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:38:59.128521 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.128499    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:38:59.179815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.179787    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume\") pod \"ffb55623-a220-41a1-804e-d88007016ee8\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") "
Sep 09 12:38:59.179961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.179837    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq756\" (UniqueName: \"kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756\") pod \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") "
Sep 09 12:38:59.179961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.179885    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxcw8\" (UniqueName: \"kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8\") pod \"ffb55623-a220-41a1-804e-d88007016ee8\" (UID: \"ffb55623-a220-41a1-804e-d88007016ee8\") "
Sep 09 12:38:59.179961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.179905    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume\") pod \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\" (UID: \"555452be-2547-4ae6-b0ac-ff3dc4312cf0\") "
Sep 09 12:38:59.180271 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.180225    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffb55623-a220-41a1-804e-d88007016ee8" (UID: "ffb55623-a220-41a1-804e-d88007016ee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:59.180468 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.180445    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "555452be-2547-4ae6-b0ac-ff3dc4312cf0" (UID: "555452be-2547-4ae6-b0ac-ff3dc4312cf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:38:59.183576 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.183548    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756" (OuterVolumeSpecName: "kube-api-access-dq756") pod "555452be-2547-4ae6-b0ac-ff3dc4312cf0" (UID: "555452be-2547-4ae6-b0ac-ff3dc4312cf0"). InnerVolumeSpecName "kube-api-access-dq756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:59.183685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.183548    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8" (OuterVolumeSpecName: "kube-api-access-wxcw8") pod "ffb55623-a220-41a1-804e-d88007016ee8" (UID: "ffb55623-a220-41a1-804e-d88007016ee8"). InnerVolumeSpecName "kube-api-access-wxcw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:59.261214 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.261189    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:38:59.281331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281233    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281304    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281336    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvpcx\" (UniqueName: \"kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281357    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281381    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281405    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir\") pod \"0d773642-185d-4ee1-a621-2036f6a2e336\" (UID: \"0d773642-185d-4ee1-a621-2036f6a2e336\") "
Sep 09 12:38:59.281798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281671    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home" (OuterVolumeSpecName: "home") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.281798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281736    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb55623-a220-41a1-804e-d88007016ee8-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.281798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281759    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq756\" (UniqueName: \"kubernetes.io/projected/555452be-2547-4ae6-b0ac-ff3dc4312cf0-kube-api-access-dq756\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.281798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281774    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcw8\" (UniqueName: \"kubernetes.io/projected/ffb55623-a220-41a1-804e-d88007016ee8-kube-api-access-wxcw8\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.281798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281787    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555452be-2547-4ae6-b0ac-ff3dc4312cf0-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.282042 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.282042 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.281911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.283944 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.283917    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm" (OuterVolumeSpecName: "dshm") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:38:59.284084 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.284022    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:38:59.284147 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.284129    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx" (OuterVolumeSpecName: "kube-api-access-fvpcx") pod "0d773642-185d-4ee1-a621-2036f6a2e336" (UID: "0d773642-185d-4ee1-a621-2036f6a2e336"). InnerVolumeSpecName "kube-api-access-fvpcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:38:59.382864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382832    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.382864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382858    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d773642-185d-4ee1-a621-2036f6a2e336-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.382864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382868    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpcx\" (UniqueName: \"kubernetes.io/projected/0d773642-185d-4ee1-a621-2036f6a2e336-kube-api-access-fvpcx\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.383079 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382876    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.383079 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382884    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.383079 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.382891    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d773642-185d-4ee1-a621-2036f6a2e336-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:38:59.853154 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.853120    2556 generic.go:356] "Generic (PLEG): container finished" podID="ffb55623-a220-41a1-804e-d88007016ee8" containerID="58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3" exitCode=0
Sep 09 12:38:59.853626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.853176    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"
Sep 09 12:38:59.853626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.853209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" event={"ID":"ffb55623-a220-41a1-804e-d88007016ee8","Type":"ContainerDied","Data":"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"}
Sep 09 12:38:59.853626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.853296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h" event={"ID":"ffb55623-a220-41a1-804e-d88007016ee8","Type":"ContainerDied","Data":"e195e952a276f8d4aeac7613292338f26c826383a66db0be361d3f7c79a7d401"}
Sep 09 12:38:59.853626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.853332    2556 scope.go:117] "RemoveContainer" containerID="58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"
Sep 09 12:38:59.854624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.854499    2556 generic.go:356] "Generic (PLEG): container finished" podID="0d773642-185d-4ee1-a621-2036f6a2e336" containerID="cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6" exitCode=0
Sep 09 12:38:59.854624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.854542    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"
Sep 09 12:38:59.854624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.854579    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" event={"ID":"0d773642-185d-4ee1-a621-2036f6a2e336","Type":"ContainerDied","Data":"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"}
Sep 09 12:38:59.854624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.854614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn" event={"ID":"0d773642-185d-4ee1-a621-2036f6a2e336","Type":"ContainerDied","Data":"9cf865f8dfe672d6d3a1309e797c1ba44b084349e1ea7a54730638d536a06838"}
Sep 09 12:38:59.856158 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.856130    2556 generic.go:356] "Generic (PLEG): container finished" podID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" containerID="7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e" exitCode=0
Sep 09 12:38:59.856291 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.856160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" event={"ID":"555452be-2547-4ae6-b0ac-ff3dc4312cf0","Type":"ContainerDied","Data":"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"}
Sep 09 12:38:59.856291 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.856183    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"
Sep 09 12:38:59.856291 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.856217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z" event={"ID":"555452be-2547-4ae6-b0ac-ff3dc4312cf0","Type":"ContainerDied","Data":"69ab5d7821e079b46a3e9f4534bc086405b2fa565f84e27111d19a341b4c1c9a"}
Sep 09 12:38:59.863439 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.863422    2556 scope.go:117] "RemoveContainer" containerID="58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"
Sep 09 12:38:59.863744 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:59.863726    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3\": container with ID starting with 58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3 not found: ID does not exist" containerID="58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"
Sep 09 12:38:59.863815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.863751    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3"} err="failed to get container status \"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3\": rpc error: code = NotFound desc = could not find container \"58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3\": container with ID starting with 58b97eba5a177dadd9647997b901e694594594c8c4af06c8584ac061ccc463f3 not found: ID does not exist"
Sep 09 12:38:59.863815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.863768    2556 scope.go:117] "RemoveContainer" containerID="cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"
Sep 09 12:38:59.871575 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.871562    2556 scope.go:117] "RemoveContainer" containerID="cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"
Sep 09 12:38:59.871821 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:59.871805    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6\": container with ID starting with cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6 not found: ID does not exist" containerID="cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"
Sep 09 12:38:59.871864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.871827    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6"} err="failed to get container status \"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6\": rpc error: code = NotFound desc = could not find container \"cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6\": container with ID starting with cc71d0cd5fa3ec177b5ed0da81067b7a95b07d7ed80ff565e9e5193ab97fdeb6 not found: ID does not exist"
Sep 09 12:38:59.871864 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.871841    2556 scope.go:117] "RemoveContainer" containerID="7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"
Sep 09 12:38:59.878119 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.878098    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:38:59.879183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.879168    2556 scope.go:117] "RemoveContainer" containerID="7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"
Sep 09 12:38:59.879437 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:38:59.879422    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e\": container with ID starting with 7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e not found: ID does not exist" containerID="7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"
Sep 09 12:38:59.879502 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.879444    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e"} err="failed to get container status \"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e\": rpc error: code = NotFound desc = could not find container \"7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e\": container with ID starting with 7fd4296d3a667d5b97d83997e7362489b05ad4e303af309f24fc3c05bdea552e not found: ID does not exist"
Sep 09 12:38:59.883449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.883424    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4ec60c-67f54c848b-cds6h"]
Sep 09 12:38:59.895455 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.895434    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:38:59.901875 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.901849    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb55623-a220-41a1-804e-d88007016ee8" path="/var/lib/kubelet/pods/ffb55623-a220-41a1-804e-d88007016ee8/volumes"
Sep 09 12:38:59.902755 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.902741    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ec60c-dfdc4dc57-tq28z"]
Sep 09 12:38:59.917173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.917144    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:38:59.922001 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:38:59.921981    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/auth-test-model-cfcdfd-kserve-7cccdcb667-mpbxn"]
Sep 09 12:39:01.901488 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:01.901455    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d773642-185d-4ee1-a621-2036f6a2e336" path="/var/lib/kubelet/pods/0d773642-185d-4ee1-a621-2036f6a2e336/volumes"
Sep 09 12:39:01.901932 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:01.901821    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" path="/var/lib/kubelet/pods/555452be-2547-4ae6-b0ac-ff3dc4312cf0/volumes"
Sep 09 12:39:02.582740 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:02.582707    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:39:02.588371 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:02.588339    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:39:13.260001 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.259969    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"]
Sep 09 12:39:13.265087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.265060    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc0270-29815950-xgkbt"]
Sep 09 12:39:13.290393 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.290363    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:39:13.290634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.290606    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" podUID="212aeda0-e271-4330-8084-07a9e0b3a0d1" containerName="payload-processing" containerID="cri-o://8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92" gracePeriod=30
Sep 09 12:39:13.540771 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.540744    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:39:13.600685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.600650    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9stth\" (UniqueName: \"kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth\") pod \"212aeda0-e271-4330-8084-07a9e0b3a0d1\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") "
Sep 09 12:39:13.600870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.600721    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume\") pod \"212aeda0-e271-4330-8084-07a9e0b3a0d1\" (UID: \"212aeda0-e271-4330-8084-07a9e0b3a0d1\") "
Sep 09 12:39:13.601066 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.601043    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "212aeda0-e271-4330-8084-07a9e0b3a0d1" (UID: "212aeda0-e271-4330-8084-07a9e0b3a0d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:39:13.602938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.602909    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth" (OuterVolumeSpecName: "kube-api-access-9stth") pod "212aeda0-e271-4330-8084-07a9e0b3a0d1" (UID: "212aeda0-e271-4330-8084-07a9e0b3a0d1"). InnerVolumeSpecName "kube-api-access-9stth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:13.701852 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.701817    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212aeda0-e271-4330-8084-07a9e0b3a0d1-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:13.701852 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.701846    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9stth\" (UniqueName: \"kubernetes.io/projected/212aeda0-e271-4330-8084-07a9e0b3a0d1-kube-api-access-9stth\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:13.901943 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.901864    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e971febc-8717-446d-a736-7b19271326c1" path="/var/lib/kubelet/pods/e971febc-8717-446d-a736-7b19271326c1/volumes"
Sep 09 12:39:13.931365 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.931334    2556 generic.go:356] "Generic (PLEG): container finished" podID="212aeda0-e271-4330-8084-07a9e0b3a0d1" containerID="8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92" exitCode=0
Sep 09 12:39:13.931508 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.931392    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"
Sep 09 12:39:13.931508 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.931423    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" event={"ID":"212aeda0-e271-4330-8084-07a9e0b3a0d1","Type":"ContainerDied","Data":"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"}
Sep 09 12:39:13.931508 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.931477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc" event={"ID":"212aeda0-e271-4330-8084-07a9e0b3a0d1","Type":"ContainerDied","Data":"d813e34e1274a24861b8fccf3d1b4bf05117b008cf862f741aed0ff646c239aa"}
Sep 09 12:39:13.931508 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.931502    2556 scope.go:117] "RemoveContainer" containerID="8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"
Sep 09 12:39:13.939742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.939724    2556 scope.go:117] "RemoveContainer" containerID="8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"
Sep 09 12:39:13.940012 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:39:13.939992    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92\": container with ID starting with 8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92 not found: ID does not exist" containerID="8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"
Sep 09 12:39:13.940063 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.940021    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92"} err="failed to get container status \"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92\": rpc error: code = NotFound desc = could not find container \"8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92\": container with ID starting with 8ab272a38badb8df033dcb436a4b5ce16e0ae5257c9b1b25b854d6366a260e92 not found: ID does not exist"
Sep 09 12:39:13.948280 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.948258    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:39:13.955433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:13.955414    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bc0270-6cd7cc84dc-8szsc"]
Sep 09 12:39:15.902087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:15.902051    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212aeda0-e271-4330-8084-07a9e0b3a0d1" path="/var/lib/kubelet/pods/212aeda0-e271-4330-8084-07a9e0b3a0d1/volumes"
Sep 09 12:39:17.602421 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.602364    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" podUID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" containerName="main" containerID="cri-o://6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8" gracePeriod=45
Sep 09 12:39:17.607258 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.607210    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" podUID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" containerName="main" containerID="cri-o://96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8" gracePeriod=45
Sep 09 12:39:17.877433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.877412    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:39:17.880089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.880071    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:39:17.938354 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938327    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2qcx\" (UniqueName: \"kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.938531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938381    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.938531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938414    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.938531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938435    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.938531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938483    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.938531 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938520    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938594    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938694    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938712    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home" (OuterVolumeSpecName: "home") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938731    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home" (OuterVolumeSpecName: "home") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.938788 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938721    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bswk\" (UniqueName: \"kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.939094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938822    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs\") pod \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\" (UID: \"86f893f2-bc6f-4afe-b71c-4348d9a0fa04\") "
Sep 09 12:39:17.939094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938860    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs\") pod \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\" (UID: \"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb\") "
Sep 09 12:39:17.939094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.938959    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache" (OuterVolumeSpecName: "model-cache") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.939278 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939114    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.939497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939289    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:17.939497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939307    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:17.939497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939320    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:17.939497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939333    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:17.939746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.939656    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.940332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.940301    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.941298 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.941233    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx" (OuterVolumeSpecName: "kube-api-access-x2qcx") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "kube-api-access-x2qcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:17.941436 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.941407    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk" (OuterVolumeSpecName: "kube-api-access-4bswk") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "kube-api-access-4bswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:39:17.941682 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.941661    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:39:17.942111 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.942089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:39:17.942336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.942317    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm" (OuterVolumeSpecName: "dshm") pod "86f893f2-bc6f-4afe-b71c-4348d9a0fa04" (UID: "86f893f2-bc6f-4afe-b71c-4348d9a0fa04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.942403 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.942337    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm" (OuterVolumeSpecName: "dshm") pod "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" (UID: "c71e2bc3-ef89-4f0d-8921-56f6ad445bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:39:17.956679 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.956648    2556 generic.go:356] "Generic (PLEG): container finished" podID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" containerID="6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8" exitCode=0
Sep 09 12:39:17.956790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.956702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"
Sep 09 12:39:17.956790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.956728    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" event={"ID":"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb","Type":"ContainerDied","Data":"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"}
Sep 09 12:39:17.956790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.956766    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn" event={"ID":"c71e2bc3-ef89-4f0d-8921-56f6ad445bcb","Type":"ContainerDied","Data":"ce0023187dd59445f81bf8e032948ae02bc4956564e91d508b95fefb8877a590"}
Sep 09 12:39:17.956790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.956782    2556 scope.go:117] "RemoveContainer" containerID="6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"
Sep 09 12:39:17.958286 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.958260    2556 generic.go:356] "Generic (PLEG): container finished" podID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" containerID="96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8" exitCode=0
Sep 09 12:39:17.958441 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.958337    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"
Sep 09 12:39:17.958550 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.958350    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" event={"ID":"86f893f2-bc6f-4afe-b71c-4348d9a0fa04","Type":"ContainerDied","Data":"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"}
Sep 09 12:39:17.958598 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.958554    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4" event={"ID":"86f893f2-bc6f-4afe-b71c-4348d9a0fa04","Type":"ContainerDied","Data":"cf27dbbc7335c8ab30d2db48b65e8249b8d7439a302ce72099836f2505ec1408"}
Sep 09 12:39:17.967408 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.967166    2556 scope.go:117] "RemoveContainer" containerID="6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"
Sep 09 12:39:17.967705 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:39:17.967673    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8\": container with ID starting with 6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8 not found: ID does not exist" containerID="6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"
Sep 09 12:39:17.967817 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.967715    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8"} err="failed to get container status \"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8\": rpc error: code = NotFound desc = could not find container \"6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8\": container with ID starting with 6dc4477fefe0a3321c1582a0d374a76e72dceba8c045a920f101202d9b821ec8 not found: ID does not exist"
Sep 09 12:39:17.967817 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.967737    2556 scope.go:117] "RemoveContainer" containerID="96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"
Sep 09 12:39:17.978781 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.978754    2556 scope.go:117] "RemoveContainer" containerID="96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"
Sep 09 12:39:17.979105 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:39:17.979086    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8\": container with ID starting with 96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8 not found: ID does not exist" containerID="96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"
Sep 09 12:39:17.979183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.979113    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8"} err="failed to get container status \"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8\": rpc error: code = NotFound desc = could not find container \"96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8\": container with ID starting with 96d784dc4b008557ae16da9b69cef6c48337fbce4ab35193a6af44f58341a2d8 not found: ID does not exist"
Sep 09 12:39:17.983532 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.983507    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:39:17.985168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.985143    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/distinct-w4-bc0270-kserve-544b6db76f-gfknn"]
Sep 09 12:39:17.996976 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.996955    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:39:18.000003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:17.999982    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc0270/premium-w4-bc0270-kserve-6c6b8dff58-hmss4"]
Sep 09 12:39:18.040470 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040443    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040492    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bswk\" (UniqueName: \"kubernetes.io/projected/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-kube-api-access-4bswk\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040503    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040514    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040521    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2qcx\" (UniqueName: \"kubernetes.io/projected/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-kube-api-access-x2qcx\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040551    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040561    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.040594 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.040570    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f893f2-bc6f-4afe-b71c-4348d9a0fa04-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:39:18.246869 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:18.246846    2556 scope.go:117] "RemoveContainer" containerID="c9aed67b54b4a8e97e31c7c5d63d9c71373266ae837fcd54b646521f4a6004b1"
Sep 09 12:39:19.902465 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:19.902422    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" path="/var/lib/kubelet/pods/86f893f2-bc6f-4afe-b71c-4348d9a0fa04/volumes"
Sep 09 12:39:19.903050 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:19.902974    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" path="/var/lib/kubelet/pods/c71e2bc3-ef89-4f0d-8921-56f6ad445bcb/volumes"
Sep 09 12:39:44.705677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.705596    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706043    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d40b59-5e24-46ef-85dd-7f85bd902430" containerName="payload-pre-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706061    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d40b59-5e24-46ef-85dd-7f85bd902430" containerName="payload-pre-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706073    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" containerName="main"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706080    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" containerName="main"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706093    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb55623-a220-41a1-804e-d88007016ee8" containerName="payload-pre-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706101    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb55623-a220-41a1-804e-d88007016ee8" containerName="payload-pre-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706109    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" containerName="payload-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706118    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" containerName="payload-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706134    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab79640-4126-45f8-8e05-7985a98cad10" containerName="payload-processing"
Sep 09 12:39:44.706136 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706141    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab79640-4126-45f8-8e05-7985a98cad10" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706151    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706158    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706172    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d773642-185d-4ee1-a621-2036f6a2e336" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706180    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d773642-185d-4ee1-a621-2036f6a2e336" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706188    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706198    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706211    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212aeda0-e271-4330-8084-07a9e0b3a0d1" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706219    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="212aeda0-e271-4330-8084-07a9e0b3a0d1" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706230    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706261    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706270    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad98c68-276b-4aee-8282-dbb70fca298c" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706277    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad98c68-276b-4aee-8282-dbb70fca298c" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706368    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab79640-4126-45f8-8e05-7985a98cad10" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706381    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d40b59-5e24-46ef-85dd-7f85bd902430" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706391    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5defbf6-c1fc-471a-b937-b5bdc2cd6b6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706400    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21eadcf-fc29-4116-9847-7ce11d3fcc6e" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706409    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="212aeda0-e271-4330-8084-07a9e0b3a0d1" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706416    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="555452be-2547-4ae6-b0ac-ff3dc4312cf0" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706423    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d773642-185d-4ee1-a621-2036f6a2e336" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706431    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad98c68-276b-4aee-8282-dbb70fca298c" containerName="payload-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706440    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb55623-a220-41a1-804e-d88007016ee8" containerName="payload-pre-processing"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706449    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f893f2-bc6f-4afe-b71c-4348d9a0fa04" containerName="main"
Sep 09 12:39:44.706654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.706456    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71e2bc3-ef89-4f0d-8921-56f6ad445bcb" containerName="main"
Sep 09 12:39:44.711520 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.711502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.715123 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.715102    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"default-dockercfg-cp998\""
Sep 09 12:39:44.715320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.715136    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"sub-test-model-cfcdfd-kserve-self-signed-certs\""
Sep 09 12:39:44.715397 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.715381    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"openshift-service-ca.crt\""
Sep 09 12:39:44.715397 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.715391    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"kube-root-ca.crt\""
Sep 09 12:39:44.721501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.721479    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:39:44.874433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874394    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.874433 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.874708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874522    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv26q\" (UniqueName: \"kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.874708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.874708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874586    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.874708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.874628    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975381 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv26q\" (UniqueName: \"kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975450    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975548    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.975949 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.976028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.976028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.975977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.977789 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.977767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.977959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.977937    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:44.988017 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:44.987991    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv26q\" (UniqueName: \"kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q\") pod \"sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:45.021077 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:45.021045    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:45.297118 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:45.297034    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:39:45.299972 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:39:45.299938    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38104d71_3b4f_4e6d_97cb_cce24047c3dc.slice/crio-c4fd15276ad30b92b992573d4ea53a1d406dfee60208bcd939ddc06960ed8e14 WatchSource:0}: Error finding container c4fd15276ad30b92b992573d4ea53a1d406dfee60208bcd939ddc06960ed8e14: Status 404 returned error can't find the container with id c4fd15276ad30b92b992573d4ea53a1d406dfee60208bcd939ddc06960ed8e14
Sep 09 12:39:46.089003 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:46.088962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" event={"ID":"38104d71-3b4f-4e6d-97cb-cce24047c3dc","Type":"ContainerStarted","Data":"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"}
Sep 09 12:39:46.089397 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:46.089013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" event={"ID":"38104d71-3b4f-4e6d-97cb-cce24047c3dc","Type":"ContainerStarted","Data":"c4fd15276ad30b92b992573d4ea53a1d406dfee60208bcd939ddc06960ed8e14"}
Sep 09 12:39:46.109170 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:46.109119    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" podStartSLOduration=2.109104132 podStartE2EDuration="2.109104132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:39:46.107816386 +0000 UTC m=+1708.738404151" watchObservedRunningTime="2026-09-09 12:39:46.109104132 +0000 UTC m=+1708.739691906"
Sep 09 12:39:55.021918 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:55.021883    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:55.022494 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:55.021963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:55.037654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:55.037627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:39:55.140180 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:39:55.140148    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:40:23.858519 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:23.858486    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7884-d2dfd4"]
Sep 09 12:40:23.861454 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:23.861438    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:23.863995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:23.863975    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kwwl8\""
Sep 09 12:40:23.870154 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:23.870133    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7884-d2dfd4"]
Sep 09 12:40:24.021233 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.021194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwl6f\" (UniqueName: \"kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f\") pod \"mt-curl-7884-d2dfd4\" (UID: \"75e59084-0387-426c-9068-6f1d8fac143e\") " pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:24.122563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.122484    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwl6f\" (UniqueName: \"kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f\") pod \"mt-curl-7884-d2dfd4\" (UID: \"75e59084-0387-426c-9068-6f1d8fac143e\") " pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:24.132478 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.132445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwl6f\" (UniqueName: \"kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f\") pod \"mt-curl-7884-d2dfd4\" (UID: \"75e59084-0387-426c-9068-6f1d8fac143e\") " pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:24.171438 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.171405    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:24.444706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.444681    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7884-d2dfd4"]
Sep 09 12:40:24.446746 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:40:24.446715    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e59084_0387_426c_9068_6f1d8fac143e.slice/crio-ffcf86ddaa3ca2269f1d7bcd86d1bdb1b326e121deb6096838676cf85025f07c WatchSource:0}: Error finding container ffcf86ddaa3ca2269f1d7bcd86d1bdb1b326e121deb6096838676cf85025f07c: Status 404 returned error can't find the container with id ffcf86ddaa3ca2269f1d7bcd86d1bdb1b326e121deb6096838676cf85025f07c
Sep 09 12:40:24.448169 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:24.448144    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:40:25.263311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:25.263280    2556 generic.go:356] "Generic (PLEG): container finished" podID="75e59084-0387-426c-9068-6f1d8fac143e" containerID="0ab8bf76c9d2b7cd881864298c923f5ee8d38cc2ba9e24e0065a639bd11b929f" exitCode=0
Sep 09 12:40:25.263674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:25.263335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7884-d2dfd4" event={"ID":"75e59084-0387-426c-9068-6f1d8fac143e","Type":"ContainerDied","Data":"0ab8bf76c9d2b7cd881864298c923f5ee8d38cc2ba9e24e0065a639bd11b929f"}
Sep 09 12:40:25.263674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:25.263364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7884-d2dfd4" event={"ID":"75e59084-0387-426c-9068-6f1d8fac143e","Type":"ContainerStarted","Data":"ffcf86ddaa3ca2269f1d7bcd86d1bdb1b326e121deb6096838676cf85025f07c"}
Sep 09 12:40:26.405535 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.405510    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:26.445260 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.445220    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7884-d2dfd4_75e59084-0387-426c-9068-6f1d8fac143e/mt-curl-7884-d2dfd4/0.log"
Sep 09 12:40:26.469230 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.469206    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7884-d2dfd4"]
Sep 09 12:40:26.473992 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.473971    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7884-d2dfd4"]
Sep 09 12:40:26.545736 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.545653    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwl6f\" (UniqueName: \"kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f\") pod \"75e59084-0387-426c-9068-6f1d8fac143e\" (UID: \"75e59084-0387-426c-9068-6f1d8fac143e\") "
Sep 09 12:40:26.547884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.547857    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f" (OuterVolumeSpecName: "kube-api-access-vwl6f") pod "75e59084-0387-426c-9068-6f1d8fac143e" (UID: "75e59084-0387-426c-9068-6f1d8fac143e"). InnerVolumeSpecName "kube-api-access-vwl6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:40:26.646497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:26.646461    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwl6f\" (UniqueName: \"kubernetes.io/projected/75e59084-0387-426c-9068-6f1d8fac143e-kube-api-access-vwl6f\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:27.276425 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:27.276398    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcf86ddaa3ca2269f1d7bcd86d1bdb1b326e121deb6096838676cf85025f07c"
Sep 09 12:40:27.276425 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:27.276399    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:27.285968 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:40:27.285939    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7884-d2dfd4\" is forbidden: User \"system:node:ip-10-0-128-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-117.ec2.internal' and this object" podUID="75e59084-0387-426c-9068-6f1d8fac143e" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:27.900402 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:40:27.900371    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7884-d2dfd4\" is forbidden: User \"system:node:ip-10-0-128-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-117.ec2.internal' and this object" podUID="75e59084-0387-426c-9068-6f1d8fac143e" pod="opendatahub/mt-curl-7884-d2dfd4"
Sep 09 12:40:27.901893 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:27.901873    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e59084-0387-426c-9068-6f1d8fac143e" path="/var/lib/kubelet/pods/75e59084-0387-426c-9068-6f1d8fac143e/volumes"
Sep 09 12:40:29.616323 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:29.616286    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:40:44.635671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.635588    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" podUID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" containerName="main" containerID="cri-o://2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b" gracePeriod=45
Sep 09 12:40:44.887884 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.887821    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:40:44.903004 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.902973    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903135 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903022    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903135 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903058    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903135 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903088    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv26q\" (UniqueName: \"kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903135 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903122    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903166    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home\") pod \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\" (UID: \"38104d71-3b4f-4e6d-97cb-cce24047c3dc\") "
Sep 09 12:40:44.903385 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903366    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:40:44.903453 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903398    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:40:44.903453 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903427    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:44.903544 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.903523    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home" (OuterVolumeSpecName: "home") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:40:44.905657 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.905624    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:40:44.906192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.906164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q" (OuterVolumeSpecName: "kube-api-access-qv26q") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "kube-api-access-qv26q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:40:44.906406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:44.906380    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm" (OuterVolumeSpecName: "dshm") pod "38104d71-3b4f-4e6d-97cb-cce24047c3dc" (UID: "38104d71-3b4f-4e6d-97cb-cce24047c3dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:40:45.004389 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.004343    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:45.004389 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.004382    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv26q\" (UniqueName: \"kubernetes.io/projected/38104d71-3b4f-4e6d-97cb-cce24047c3dc-kube-api-access-qv26q\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:45.004389 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.004396    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:45.004634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.004408    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38104d71-3b4f-4e6d-97cb-cce24047c3dc-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:45.004634 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.004419    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38104d71-3b4f-4e6d-97cb-cce24047c3dc-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:40:45.362343 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.362310    2556 generic.go:356] "Generic (PLEG): container finished" podID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" containerID="2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b" exitCode=0
Sep 09 12:40:45.362513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.362355    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"
Sep 09 12:40:45.362513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.362399    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" event={"ID":"38104d71-3b4f-4e6d-97cb-cce24047c3dc","Type":"ContainerDied","Data":"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"}
Sep 09 12:40:45.362513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.362446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4" event={"ID":"38104d71-3b4f-4e6d-97cb-cce24047c3dc","Type":"ContainerDied","Data":"c4fd15276ad30b92b992573d4ea53a1d406dfee60208bcd939ddc06960ed8e14"}
Sep 09 12:40:45.362513 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.362468    2556 scope.go:117] "RemoveContainer" containerID="2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"
Sep 09 12:40:45.371580 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.371551    2556 scope.go:117] "RemoveContainer" containerID="2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"
Sep 09 12:40:45.371892 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:40:45.371871    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b\": container with ID starting with 2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b not found: ID does not exist" containerID="2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"
Sep 09 12:40:45.371947 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.371903    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b"} err="failed to get container status \"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b\": rpc error: code = NotFound desc = could not find container \"2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b\": container with ID starting with 2eefd2e49bbb6b44c601550b7e4f7b41e247ea76300616d04720a7b871efc83b not found: ID does not exist"
Sep 09 12:40:45.384185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.384156    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:40:45.388690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.388664    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/sub-test-model-cfcdfd-kserve-5df5fdb767-jtqm4"]
Sep 09 12:40:45.901966 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:45.901930    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" path="/var/lib/kubelet/pods/38104d71-3b4f-4e6d-97cb-cce24047c3dc/volumes"
Sep 09 12:40:58.016424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016390    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:40:58.016786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016723    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e59084-0387-426c-9068-6f1d8fac143e" containerName="mt-curl-7884-d2dfd4"
Sep 09 12:40:58.016786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016736    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e59084-0387-426c-9068-6f1d8fac143e" containerName="mt-curl-7884-d2dfd4"
Sep 09 12:40:58.016786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016746    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" containerName="main"
Sep 09 12:40:58.016786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016753    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" containerName="main"
Sep 09 12:40:58.016923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016818    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e59084-0387-426c-9068-6f1d8fac143e" containerName="mt-curl-7884-d2dfd4"
Sep 09 12:40:58.016923 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.016825    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="38104d71-3b4f-4e6d-97cb-cce24047c3dc" containerName="main"
Sep 09 12:40:58.022273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.022229    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.024971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.024947    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"default-dockercfg-cp998\""
Sep 09 12:40:58.025900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.025882    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"rate-test-model-cfcdfd-kserve-self-signed-certs\""
Sep 09 12:40:58.027818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.027798    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"kube-root-ca.crt\""
Sep 09 12:40:58.032563 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.032542    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cfcdfd\"/\"openshift-service-ca.crt\""
Sep 09 12:40:58.034711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.034688    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:40:58.115298 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.115485 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.115485 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115353    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.115485 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115400    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.115485 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.115695 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.115553    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn9g\" (UniqueName: \"kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.216942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.216897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.216951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.216985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn9g\" (UniqueName: \"kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.217682 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.217582    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.219459 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.219440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.219636 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.219615    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.226618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.226595    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn9g\" (UniqueName: \"kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g\") pod \"rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") " pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.330746 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.330652    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:40:58.605458 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:58.605435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:40:58.607741 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:40:58.607704    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3aa2d73_f68f_45fc_b46b_901ce6204c2a.slice/crio-02ad65c5750e9913246c4781cc00332defe248681ecd8599e0da608ba70d5d38 WatchSource:0}: Error finding container 02ad65c5750e9913246c4781cc00332defe248681ecd8599e0da608ba70d5d38: Status 404 returned error can't find the container with id 02ad65c5750e9913246c4781cc00332defe248681ecd8599e0da608ba70d5d38
Sep 09 12:40:59.433335 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:59.433292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" event={"ID":"d3aa2d73-f68f-45fc-b46b-901ce6204c2a","Type":"ContainerStarted","Data":"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"}
Sep 09 12:40:59.433335 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:59.433341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" event={"ID":"d3aa2d73-f68f-45fc-b46b-901ce6204c2a","Type":"ContainerStarted","Data":"02ad65c5750e9913246c4781cc00332defe248681ecd8599e0da608ba70d5d38"}
Sep 09 12:40:59.452615 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:40:59.452564    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" podStartSLOduration=2.452544284 podStartE2EDuration="2.452544284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:40:59.450658759 +0000 UTC m=+1782.081246535" watchObservedRunningTime="2026-09-09 12:40:59.452544284 +0000 UTC m=+1782.083132062"
Sep 09 12:41:08.331288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:08.331230    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:41:08.331288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:08.331296    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:41:08.343742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:08.343717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:41:08.485171 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:08.485094    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:41:17.914705 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:17.914678    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:41:17.916474 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:17.916451    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:41:17.919282 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:17.919262    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:41:17.920541 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:17.920518    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:41:18.304547 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:18.304520    2556 scope.go:117] "RemoveContainer" containerID="150381c406e0c75f17465805f6b3d7012c647843b4c049b6c64123b71cd767c3"
Sep 09 12:41:45.106341 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:45.106300    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:41:51.057677 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.057640    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:41:51.060920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.060903    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.063448 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.063420    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-232ce2\""
Sep 09 12:41:51.063609 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.063587    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-232ce2-dockercfg-qnqm4\""
Sep 09 12:41:51.071317 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.071292    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:41:51.073882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.073860    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:41:51.077422 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.077402    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.088451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.088428    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:41:51.187168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.187133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czn2\" (UniqueName: \"kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.187168 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.187172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqhb\" (UniqueName: \"kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.187411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.187208    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.187411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.187338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.288161 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.288125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.288391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.288226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czn2\" (UniqueName: \"kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.288391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.288297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqhb\" (UniqueName: \"kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.288391 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.288332    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.289013 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.288992    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.289059 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.289041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.297853 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.297820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czn2\" (UniqueName: \"kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2\") pod \"payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.297993 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.297896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqhb\" (UniqueName: \"kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb\") pod \"payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.371961 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.371851    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:51.387992 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.387880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:51.647273 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.647231    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:41:51.649729 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:41:51.649701    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25e3cb1_7a98_4b74_9fff_a1ff86af6dbc.slice/crio-7f54ac2590f76b41045e5d513d658a5ea73e04ff5eb2c482da920dd3b74c2948 WatchSource:0}: Error finding container 7f54ac2590f76b41045e5d513d658a5ea73e04ff5eb2c482da920dd3b74c2948: Status 404 returned error can't find the container with id 7f54ac2590f76b41045e5d513d658a5ea73e04ff5eb2c482da920dd3b74c2948
Sep 09 12:41:51.673116 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.673082    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:41:51.674173 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:51.674142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" event={"ID":"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc","Type":"ContainerStarted","Data":"7f54ac2590f76b41045e5d513d658a5ea73e04ff5eb2c482da920dd3b74c2948"}
Sep 09 12:41:51.678425 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:41:51.678403    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaa63fa_24f7_43f4_9c43_36a9f275325e.slice/crio-b95290244d35e677c965ed10c5053f886d5dc56735182a9ea982df5b0440cdbc WatchSource:0}: Error finding container b95290244d35e677c965ed10c5053f886d5dc56735182a9ea982df5b0440cdbc: Status 404 returned error can't find the container with id b95290244d35e677c965ed10c5053f886d5dc56735182a9ea982df5b0440cdbc
Sep 09 12:41:52.681219 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.681171    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" event={"ID":"7aaa63fa-24f7-43f4-9c43-36a9f275325e","Type":"ContainerStarted","Data":"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"}
Sep 09 12:41:52.681724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.681260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" event={"ID":"7aaa63fa-24f7-43f4-9c43-36a9f275325e","Type":"ContainerStarted","Data":"b95290244d35e677c965ed10c5053f886d5dc56735182a9ea982df5b0440cdbc"}
Sep 09 12:41:52.681724 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.681290    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:41:52.682977 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.682953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" event={"ID":"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc","Type":"ContainerStarted","Data":"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"}
Sep 09 12:41:52.683085 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.683016    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:41:52.699166 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.699114    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" podStartSLOduration=1.699096932 podStartE2EDuration="1.699096932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:41:52.697850834 +0000 UTC m=+1835.328438607" watchObservedRunningTime="2026-09-09 12:41:52.699096932 +0000 UTC m=+1835.329684740"
Sep 09 12:41:52.714315 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:41:52.714263    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" podStartSLOduration=1.714228136 podStartE2EDuration="1.714228136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:41:52.712733999 +0000 UTC m=+1835.343321775" watchObservedRunningTime="2026-09-09 12:41:52.714228136 +0000 UTC m=+1835.344815975"
Sep 09 12:42:00.126996 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.126949    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" podUID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" containerName="main" containerID="cri-o://15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d" gracePeriod=45
Sep 09 12:42:00.369284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.369261    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:42:00.475796 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475766    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.475995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475811    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.475995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475836    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.475995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475867    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.475995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475906    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.475995 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.475970    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgn9g\" (UniqueName: \"kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g\") pod \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\" (UID: \"d3aa2d73-f68f-45fc-b46b-901ce6204c2a\") "
Sep 09 12:42:00.476305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.476157    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:42:00.476305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.476167    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home" (OuterVolumeSpecName: "home") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:42:00.476305 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.476197    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:42:00.478219 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.478189    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm" (OuterVolumeSpecName: "dshm") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:42:00.478344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.478278    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:42:00.478344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.478296    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g" (OuterVolumeSpecName: "kube-api-access-pgn9g") pod "d3aa2d73-f68f-45fc-b46b-901ce6204c2a" (UID: "d3aa2d73-f68f-45fc-b46b-901ce6204c2a"). InnerVolumeSpecName "kube-api-access-pgn9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:42:00.577431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577382    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.577431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577421    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgn9g\" (UniqueName: \"kubernetes.io/projected/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-kube-api-access-pgn9g\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.577431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577436    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.577675 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577448    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.577675 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577458    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.577675 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.577469    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3aa2d73-f68f-45fc-b46b-901ce6204c2a-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:42:00.721684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.721651    2556 generic.go:356] "Generic (PLEG): container finished" podID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" containerID="15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d" exitCode=0
Sep 09 12:42:00.721877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.721698    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"
Sep 09 12:42:00.721877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.721734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" event={"ID":"d3aa2d73-f68f-45fc-b46b-901ce6204c2a","Type":"ContainerDied","Data":"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"}
Sep 09 12:42:00.721877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.721777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg" event={"ID":"d3aa2d73-f68f-45fc-b46b-901ce6204c2a","Type":"ContainerDied","Data":"02ad65c5750e9913246c4781cc00332defe248681ecd8599e0da608ba70d5d38"}
Sep 09 12:42:00.721877 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.721795    2556 scope.go:117] "RemoveContainer" containerID="15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"
Sep 09 12:42:00.730633 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.730616    2556 scope.go:117] "RemoveContainer" containerID="15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"
Sep 09 12:42:00.730879 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:42:00.730862    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d\": container with ID starting with 15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d not found: ID does not exist" containerID="15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"
Sep 09 12:42:00.730938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.730888    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d"} err="failed to get container status \"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d\": rpc error: code = NotFound desc = could not find container \"15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d\": container with ID starting with 15ef8f2b0ace7ea50b4b35d78a599fa2ff7fbd0d4c0e9185d82604774f106f8d not found: ID does not exist"
Sep 09 12:42:00.743851 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.743829    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:42:00.751818 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:00.751799    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cfcdfd/rate-test-model-cfcdfd-kserve-6f8c5d5f9-pm4qg"]
Sep 09 12:42:01.901798 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:01.901767    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" path="/var/lib/kubelet/pods/d3aa2d73-f68f-45fc-b46b-901ce6204c2a/volumes"
Sep 09 12:42:03.690126 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:03.690091    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:42:03.690564 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:03.690215    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:42:20.150700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.150662    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:42:20.151113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.151010    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" containerName="main"
Sep 09 12:42:20.151113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.151022    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" containerName="main"
Sep 09 12:42:20.151113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.151078    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3aa2d73-f68f-45fc-b46b-901ce6204c2a" containerName="main"
Sep 09 12:42:20.153900 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.153876    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.156356 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.156334    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8ec58b\""
Sep 09 12:42:20.156486 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.156337    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8ec58b-dockercfg-7cqn4\""
Sep 09 12:42:20.163773 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.163748    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:42:20.210068 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.210033    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:42:20.213192 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.213169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.225705 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.225673    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:42:20.254880 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.254844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.255050 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.254991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwmg\" (UniqueName: \"kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.355863 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.355828    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwmg\" (UniqueName: \"kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.356069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.355911    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkh8\" (UniqueName: \"kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.356069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.355934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.356069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.355965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.356703 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.356678    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.364206 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.364182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwmg\" (UniqueName: \"kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg\") pod \"payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.456683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.456591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkh8\" (UniqueName: \"kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.456683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.456633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.457394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.457369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.466313 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.466289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkh8\" (UniqueName: \"kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8\") pod \"payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.531224 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.531179    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:20.765424 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.765378    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:20.815428 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:20.814765    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:42:21.039094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.039068    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:42:21.041101 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:42:21.041071    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ee1aed_f0b0_4808_991f_d4fbd37749a4.slice/crio-d521e9756242cefc79390635f01a535ca3478a5c40adb299c814c9f5f123194e WatchSource:0}: Error finding container d521e9756242cefc79390635f01a535ca3478a5c40adb299c814c9f5f123194e: Status 404 returned error can't find the container with id d521e9756242cefc79390635f01a535ca3478a5c40adb299c814c9f5f123194e
Sep 09 12:42:21.828264 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.828120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" event={"ID":"e1be66d4-1a63-45e2-abdb-7a18ce706ff9","Type":"ContainerStarted","Data":"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"}
Sep 09 12:42:21.828264 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.828186    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:21.828264 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.828210    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" event={"ID":"e1be66d4-1a63-45e2-abdb-7a18ce706ff9","Type":"ContainerStarted","Data":"b61d89adaa61d7a01166cd756ae15ad4188bc67ddf89d256cc2b869b98e1cd54"}
Sep 09 12:42:21.830147 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.830122    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" event={"ID":"67ee1aed-f0b0-4808-991f-d4fbd37749a4","Type":"ContainerStarted","Data":"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"}
Sep 09 12:42:21.830299 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.830169    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" event={"ID":"67ee1aed-f0b0-4808-991f-d4fbd37749a4","Type":"ContainerStarted","Data":"d521e9756242cefc79390635f01a535ca3478a5c40adb299c814c9f5f123194e"}
Sep 09 12:42:21.830299 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.830198    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:21.845978 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.845923    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" podStartSLOduration=1.845905799 podStartE2EDuration="1.845905799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:42:21.845370485 +0000 UTC m=+1864.475958261" watchObservedRunningTime="2026-09-09 12:42:21.845905799 +0000 UTC m=+1864.476493576"
Sep 09 12:42:21.861060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:21.861013    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" podStartSLOduration=1.860998168 podStartE2EDuration="1.860998168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:42:21.860688692 +0000 UTC m=+1864.491276472" watchObservedRunningTime="2026-09-09 12:42:21.860998168 +0000 UTC m=+1864.491585944"
Sep 09 12:42:32.837113 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:32.837079    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:42:32.837566 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:32.837184    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:42:48.751475 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:42:48.751387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2_67ee1aed-f0b0-4808-991f-d4fbd37749a4/payload-processing/0.log"
Sep 09 12:43:19.630296 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:19.630258    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z_7aaa63fa-24f7-43f4-9c43-36a9f275325e/payload-processing/0.log"
Sep 09 12:43:19.894257 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:19.894166    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2_67ee1aed-f0b0-4808-991f-d4fbd37749a4/payload-processing/0.log"
Sep 09 12:43:25.536983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.536942    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:25.540636 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.540613    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.543215 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.543180    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-330769\""
Sep 09 12:43:25.543356 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.543226    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-330769-dockercfg-grkv2\""
Sep 09 12:43:25.550830 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.550806    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:25.575481 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.575452    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:25.578714 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.578699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.592308 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.592287    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:25.625179 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.625146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.625387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.625270    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.625387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.625309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8gw\" (UniqueName: \"kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.625387 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.625337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdzq\" (UniqueName: \"kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.726088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.726088 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8gw\" (UniqueName: \"kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.726327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdzq\" (UniqueName: \"kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.726327 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.726865 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.726913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.726895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.735991 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.735965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8gw\" (UniqueName: \"kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw\") pod \"payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.736959 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.736931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdzq\" (UniqueName: \"kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:25.851009 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.850937    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:25.888363 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:25.888320    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:26.130494 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:26.130472    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:26.132494 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:43:26.132462    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96d53b7_aa7d_429d_9a4c_7bb981bea218.slice/crio-2902eb7a8eb564d6a053d5ae85a61e741fb95c1344c182b0548c5f3a2434cf8d WatchSource:0}: Error finding container 2902eb7a8eb564d6a053d5ae85a61e741fb95c1344c182b0548c5f3a2434cf8d: Status 404 returned error can't find the container with id 2902eb7a8eb564d6a053d5ae85a61e741fb95c1344c182b0548c5f3a2434cf8d
Sep 09 12:43:26.176018 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:26.175986    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:26.178865 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:43:26.178842    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96364e1a_e1b7_4c13_b399_8b792a195125.slice/crio-13a8bcefd52f472d3528aa71d66871b72664ed64fdfc46ef42ca38c4c142788c WatchSource:0}: Error finding container 13a8bcefd52f472d3528aa71d66871b72664ed64fdfc46ef42ca38c4c142788c: Status 404 returned error can't find the container with id 13a8bcefd52f472d3528aa71d66871b72664ed64fdfc46ef42ca38c4c142788c
Sep 09 12:43:27.127499 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.127462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" event={"ID":"96364e1a-e1b7-4c13-b399-8b792a195125","Type":"ContainerStarted","Data":"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"}
Sep 09 12:43:27.128008 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.127526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" event={"ID":"96364e1a-e1b7-4c13-b399-8b792a195125","Type":"ContainerStarted","Data":"13a8bcefd52f472d3528aa71d66871b72664ed64fdfc46ef42ca38c4c142788c"}
Sep 09 12:43:27.128008 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.127555    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:27.129207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.129179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" event={"ID":"a96d53b7-aa7d-429d-9a4c-7bb981bea218","Type":"ContainerStarted","Data":"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"}
Sep 09 12:43:27.129358 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.129220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" event={"ID":"a96d53b7-aa7d-429d-9a4c-7bb981bea218","Type":"ContainerStarted","Data":"2902eb7a8eb564d6a053d5ae85a61e741fb95c1344c182b0548c5f3a2434cf8d"}
Sep 09 12:43:27.129358 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.129272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:27.156396 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.156349    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" podStartSLOduration=2.156333127 podStartE2EDuration="2.156333127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:43:27.154883356 +0000 UTC m=+1929.785471131" watchObservedRunningTime="2026-09-09 12:43:27.156333127 +0000 UTC m=+1929.786920945"
Sep 09 12:43:27.177227 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:27.177181    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" podStartSLOduration=2.17716548 podStartE2EDuration="2.17716548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:43:27.175043502 +0000 UTC m=+1929.805631278" watchObservedRunningTime="2026-09-09 12:43:27.17716548 +0000 UTC m=+1929.807753255"
Sep 09 12:43:38.136204 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:38.136170    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:38.136619 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:38.136278    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:55.637191 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.637151    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:55.637761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.637734    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" podUID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" containerName="payload-processing" containerID="cri-o://0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb" gracePeriod=30
Sep 09 12:43:55.648540 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.648513    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:55.648761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.648718    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" podUID="96364e1a-e1b7-4c13-b399-8b792a195125" containerName="payload-pre-processing" containerID="cri-o://4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114" gracePeriod=30
Sep 09 12:43:55.918082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.918056    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:55.921430 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:55.921407    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:56.095833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.095737    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume\") pod \"96364e1a-e1b7-4c13-b399-8b792a195125\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") "
Sep 09 12:43:56.095833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.095798    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume\") pod \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") "
Sep 09 12:43:56.095833 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.095826    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsdzq\" (UniqueName: \"kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq\") pod \"96364e1a-e1b7-4c13-b399-8b792a195125\" (UID: \"96364e1a-e1b7-4c13-b399-8b792a195125\") "
Sep 09 12:43:56.096127 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.095875    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw8gw\" (UniqueName: \"kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw\") pod \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\" (UID: \"a96d53b7-aa7d-429d-9a4c-7bb981bea218\") "
Sep 09 12:43:56.096199 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.096173    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a96d53b7-aa7d-429d-9a4c-7bb981bea218" (UID: "a96d53b7-aa7d-429d-9a4c-7bb981bea218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:43:56.096282 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.096200    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96364e1a-e1b7-4c13-b399-8b792a195125" (UID: "96364e1a-e1b7-4c13-b399-8b792a195125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:43:56.097988 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.097967    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq" (OuterVolumeSpecName: "kube-api-access-dsdzq") pod "96364e1a-e1b7-4c13-b399-8b792a195125" (UID: "96364e1a-e1b7-4c13-b399-8b792a195125"). InnerVolumeSpecName "kube-api-access-dsdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:43:56.098051 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.098023    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw" (OuterVolumeSpecName: "kube-api-access-hw8gw") pod "a96d53b7-aa7d-429d-9a4c-7bb981bea218" (UID: "a96d53b7-aa7d-429d-9a4c-7bb981bea218"). InnerVolumeSpecName "kube-api-access-hw8gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:43:56.196861 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.196824    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96364e1a-e1b7-4c13-b399-8b792a195125-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:43:56.196861 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.196852    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d53b7-aa7d-429d-9a4c-7bb981bea218-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:43:56.196861 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.196862    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdzq\" (UniqueName: \"kubernetes.io/projected/96364e1a-e1b7-4c13-b399-8b792a195125-kube-api-access-dsdzq\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:43:56.197086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.196871    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8gw\" (UniqueName: \"kubernetes.io/projected/a96d53b7-aa7d-429d-9a4c-7bb981bea218-kube-api-access-hw8gw\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:43:56.272101 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.272062    2556 generic.go:356] "Generic (PLEG): container finished" podID="96364e1a-e1b7-4c13-b399-8b792a195125" containerID="4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114" exitCode=0
Sep 09 12:43:56.272332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.272109    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"
Sep 09 12:43:56.272332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.272152    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" event={"ID":"96364e1a-e1b7-4c13-b399-8b792a195125","Type":"ContainerDied","Data":"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"}
Sep 09 12:43:56.272332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.272208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4" event={"ID":"96364e1a-e1b7-4c13-b399-8b792a195125","Type":"ContainerDied","Data":"13a8bcefd52f472d3528aa71d66871b72664ed64fdfc46ef42ca38c4c142788c"}
Sep 09 12:43:56.272332 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.272267    2556 scope.go:117] "RemoveContainer" containerID="4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"
Sep 09 12:43:56.274026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.273943    2556 generic.go:356] "Generic (PLEG): container finished" podID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" containerID="0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb" exitCode=0
Sep 09 12:43:56.274026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.273989    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"
Sep 09 12:43:56.274026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.273997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" event={"ID":"a96d53b7-aa7d-429d-9a4c-7bb981bea218","Type":"ContainerDied","Data":"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"}
Sep 09 12:43:56.274264 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.274035    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4" event={"ID":"a96d53b7-aa7d-429d-9a4c-7bb981bea218","Type":"ContainerDied","Data":"2902eb7a8eb564d6a053d5ae85a61e741fb95c1344c182b0548c5f3a2434cf8d"}
Sep 09 12:43:56.281560 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.281419    2556 scope.go:117] "RemoveContainer" containerID="4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"
Sep 09 12:43:56.281759 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:43:56.281731    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114\": container with ID starting with 4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114 not found: ID does not exist" containerID="4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"
Sep 09 12:43:56.281821 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.281770    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114"} err="failed to get container status \"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114\": rpc error: code = NotFound desc = could not find container \"4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114\": container with ID starting with 4eb6dde637e9e04c9eea37bcbb32f7c045c122a038cca48124e954abb8cc3114 not found: ID does not exist"
Sep 09 12:43:56.281821 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.281792    2556 scope.go:117] "RemoveContainer" containerID="0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"
Sep 09 12:43:56.289700 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.289684    2556 scope.go:117] "RemoveContainer" containerID="0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"
Sep 09 12:43:56.289963 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:43:56.289946    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb\": container with ID starting with 0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb not found: ID does not exist" containerID="0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"
Sep 09 12:43:56.290024 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.289975    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb"} err="failed to get container status \"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb\": rpc error: code = NotFound desc = could not find container \"0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb\": container with ID starting with 0a26d23f6877ff3d898b60e1bd2061d672515b5ca461408046585882d4004bdb not found: ID does not exist"
Sep 09 12:43:56.300343 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.300313    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:56.312263 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.312218    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-330769-67876799f-dm5r4"]
Sep 09 12:43:56.328082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.328061    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:56.336163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:56.336143    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-330769-6d9db4db58-xzhd4"]
Sep 09 12:43:57.901389 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:57.901357    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96364e1a-e1b7-4c13-b399-8b792a195125" path="/var/lib/kubelet/pods/96364e1a-e1b7-4c13-b399-8b792a195125/volumes"
Sep 09 12:43:57.901893 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:43:57.901870    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" path="/var/lib/kubelet/pods/a96d53b7-aa7d-429d-9a4c-7bb981bea218/volumes"
Sep 09 12:44:08.261684 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.261650    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"]
Sep 09 12:44:08.262044 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.261975    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96364e1a-e1b7-4c13-b399-8b792a195125" containerName="payload-pre-processing"
Sep 09 12:44:08.262044 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.261986    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="96364e1a-e1b7-4c13-b399-8b792a195125" containerName="payload-pre-processing"
Sep 09 12:44:08.262044 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.262005    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" containerName="payload-processing"
Sep 09 12:44:08.262044 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.262011    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" containerName="payload-processing"
Sep 09 12:44:08.262177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.262067    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96d53b7-aa7d-429d-9a4c-7bb981bea218" containerName="payload-processing"
Sep 09 12:44:08.262177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.262075    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="96364e1a-e1b7-4c13-b399-8b792a195125" containerName="payload-pre-processing"
Sep 09 12:44:08.266364 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.266346    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:08.268706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.268681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:44:08.268825 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.268773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:44:08.281084 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.281061    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"]
Sep 09 12:44:08.304915 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.304888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv\" (UID: \"6b764565-5a70-4606-9239-00487dd9125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:08.406163 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.406137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv\" (UID: \"6b764565-5a70-4606-9239-00487dd9125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:08.406761 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.406743    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv\" (UID: \"6b764565-5a70-4606-9239-00487dd9125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:08.575868 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.575755    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:08.847051 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:08.847024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"]
Sep 09 12:44:08.849168 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:44:08.849141    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b764565_5a70_4606_9239_00487dd9125e.slice/crio-4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d WatchSource:0}: Error finding container 4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d: Status 404 returned error can't find the container with id 4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d
Sep 09 12:44:09.337406 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:09.337369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv" event={"ID":"6b764565-5a70-4606-9239-00487dd9125e","Type":"ContainerStarted","Data":"7e1d07fbd0e02f975227828475180021465d2bc251dc563e0370f3eef49ed1d6"}
Sep 09 12:44:09.337793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:09.337416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv" event={"ID":"6b764565-5a70-4606-9239-00487dd9125e","Type":"ContainerStarted","Data":"4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d"}
Sep 09 12:44:09.352413 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:09.352317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv" podStartSLOduration=1.352304349 podStartE2EDuration="1.352304349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:44:09.352072682 +0000 UTC m=+1971.982660457" watchObservedRunningTime="2026-09-09 12:44:09.352304349 +0000 UTC m=+1971.982892123"
Sep 09 12:44:10.342622 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:10.342586    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b764565-5a70-4606-9239-00487dd9125e" containerID="7e1d07fbd0e02f975227828475180021465d2bc251dc563e0370f3eef49ed1d6" exitCode=0
Sep 09 12:44:10.343001 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:10.342671    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv" event={"ID":"6b764565-5a70-4606-9239-00487dd9125e","Type":"ContainerDied","Data":"7e1d07fbd0e02f975227828475180021465d2bc251dc563e0370f3eef49ed1d6"}
Sep 09 12:44:11.475943 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:11.475917    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:11.532286 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:11.532228    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca\") pod \"6b764565-5a70-4606-9239-00487dd9125e\" (UID: \"6b764565-5a70-4606-9239-00487dd9125e\") "
Sep 09 12:44:11.532623 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:11.532600    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b764565-5a70-4606-9239-00487dd9125e" (UID: "6b764565-5a70-4606-9239-00487dd9125e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:44:11.633607 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:11.633516    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b764565-5a70-4606-9239-00487dd9125e-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:44:12.353674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:12.353642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv" event={"ID":"6b764565-5a70-4606-9239-00487dd9125e","Type":"ContainerDied","Data":"4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d"}
Sep 09 12:44:12.353850 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:12.353689    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d3a029bb14609a9b86a071da30ff4c06d89d15db4e4cdfbd255ace50e92774d"
Sep 09 12:44:12.353850 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:12.353651    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"
Sep 09 12:44:23.887184 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:23.887145    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:44:23.887648 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:23.887403    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" podUID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" containerName="payload-processing" containerID="cri-o://b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9" gracePeriod=30
Sep 09 12:44:23.890882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:23.890855    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:44:23.891655 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:23.891062    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" podUID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" containerName="payload-pre-processing" containerID="cri-o://084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341" gracePeriod=30
Sep 09 12:44:24.134786 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.134755    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"]
Sep 09 12:44:24.141207 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.141140    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-232ce2-slhwv"]
Sep 09 12:44:24.175015 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.174997    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:44:24.177674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.177657    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:44:24.250837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.250788    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqhb\" (UniqueName: \"kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb\") pod \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") "
Sep 09 12:44:24.250837 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.250842    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8czn2\" (UniqueName: \"kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2\") pod \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") "
Sep 09 12:44:24.251102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.250874    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume\") pod \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\" (UID: \"7aaa63fa-24f7-43f4-9c43-36a9f275325e\") "
Sep 09 12:44:24.251102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.250902    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume\") pod \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\" (UID: \"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc\") "
Sep 09 12:44:24.251320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.251296    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aaa63fa-24f7-43f4-9c43-36a9f275325e" (UID: "7aaa63fa-24f7-43f4-9c43-36a9f275325e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:44:24.251378 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.251342    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" (UID: "a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:44:24.253120 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.253099    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb" (OuterVolumeSpecName: "kube-api-access-2cqhb") pod "a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" (UID: "a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc"). InnerVolumeSpecName "kube-api-access-2cqhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:24.253182 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.253145    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2" (OuterVolumeSpecName: "kube-api-access-8czn2") pod "7aaa63fa-24f7-43f4-9c43-36a9f275325e" (UID: "7aaa63fa-24f7-43f4-9c43-36a9f275325e"). InnerVolumeSpecName "kube-api-access-8czn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:44:24.352131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.352095    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqhb\" (UniqueName: \"kubernetes.io/projected/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-kube-api-access-2cqhb\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:44:24.352131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.352123    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czn2\" (UniqueName: \"kubernetes.io/projected/7aaa63fa-24f7-43f4-9c43-36a9f275325e-kube-api-access-8czn2\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:44:24.352131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.352133    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaa63fa-24f7-43f4-9c43-36a9f275325e-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:44:24.352131 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.352142    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:44:24.413146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.413059    2556 generic.go:356] "Generic (PLEG): container finished" podID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" containerID="b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9" exitCode=0
Sep 09 12:44:24.413146 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.413105    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"
Sep 09 12:44:24.413372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.413151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" event={"ID":"7aaa63fa-24f7-43f4-9c43-36a9f275325e","Type":"ContainerDied","Data":"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"}
Sep 09 12:44:24.413372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.413211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z" event={"ID":"7aaa63fa-24f7-43f4-9c43-36a9f275325e","Type":"ContainerDied","Data":"b95290244d35e677c965ed10c5053f886d5dc56735182a9ea982df5b0440cdbc"}
Sep 09 12:44:24.413372 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.413258    2556 scope.go:117] "RemoveContainer" containerID="b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"
Sep 09 12:44:24.414600 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.414578    2556 generic.go:356] "Generic (PLEG): container finished" podID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" containerID="084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341" exitCode=0
Sep 09 12:44:24.414711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.414626    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"
Sep 09 12:44:24.414711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.414659    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" event={"ID":"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc","Type":"ContainerDied","Data":"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"}
Sep 09 12:44:24.414711 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.414702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t" event={"ID":"a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc","Type":"ContainerDied","Data":"7f54ac2590f76b41045e5d513d658a5ea73e04ff5eb2c482da920dd3b74c2948"}
Sep 09 12:44:24.423517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.423504    2556 scope.go:117] "RemoveContainer" containerID="b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"
Sep 09 12:44:24.423803 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:44:24.423784    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9\": container with ID starting with b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9 not found: ID does not exist" containerID="b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"
Sep 09 12:44:24.423859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.423811    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9"} err="failed to get container status \"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9\": rpc error: code = NotFound desc = could not find container \"b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9\": container with ID starting with b5db526bfb5c3d43a75bdd0fc2f6472328c088ffbfb2a4a928659bbb0091b6d9 not found: ID does not exist"
Sep 09 12:44:24.423859 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.423826    2556 scope.go:117] "RemoveContainer" containerID="084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"
Sep 09 12:44:24.431499 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.431481    2556 scope.go:117] "RemoveContainer" containerID="084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"
Sep 09 12:44:24.431762 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:44:24.431742    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341\": container with ID starting with 084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341 not found: ID does not exist" containerID="084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"
Sep 09 12:44:24.431804 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.431769    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341"} err="failed to get container status \"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341\": rpc error: code = NotFound desc = could not find container \"084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341\": container with ID starting with 084bef03717c3c58b2ed4cd421bc8e736b6bc3baa22425d6d4f3a85806e2d341 not found: ID does not exist"
Sep 09 12:44:24.444324 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.444303    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:44:24.452023 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.452002    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-232ce2-8d75f8f87-rv89z"]
Sep 09 12:44:24.469530 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.469508    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:44:24.476430 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:24.476411    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-232ce2-65b9c597c8-qg86t"]
Sep 09 12:44:25.903183 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:25.903146    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b764565-5a70-4606-9239-00487dd9125e" path="/var/lib/kubelet/pods/6b764565-5a70-4606-9239-00487dd9125e/volumes"
Sep 09 12:44:25.903899 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:25.903877    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" path="/var/lib/kubelet/pods/7aaa63fa-24f7-43f4-9c43-36a9f275325e/volumes"
Sep 09 12:44:25.904434 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:44:25.904412    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" path="/var/lib/kubelet/pods/a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc/volumes"
Sep 09 12:45:00.156787 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.156752    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"]
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157120    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" containerName="payload-pre-processing"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157137    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" containerName="payload-pre-processing"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157150    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b764565-5a70-4606-9239-00487dd9125e" containerName="revoke-keys"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157158    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b764565-5a70-4606-9239-00487dd9125e" containerName="revoke-keys"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157170    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" containerName="payload-processing"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157179    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" containerName="payload-processing"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157291    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b764565-5a70-4606-9239-00487dd9125e" containerName="revoke-keys"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157303    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aaa63fa-24f7-43f4-9c43-36a9f275325e" containerName="payload-processing"
Sep 09 12:45:00.157375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.157314    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25e3cb1-7a98-4b74-9fff-a1ff86af6dbc" containerName="payload-pre-processing"
Sep 09 12:45:00.160328 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.160310    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"
Sep 09 12:45:00.167150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.166925    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:45:00.168602 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.168580    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"]
Sep 09 12:45:00.170226 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.170207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"
Sep 09 12:45:00.171951 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.171931    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"
Sep 09 12:45:00.172856 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.172831    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"]
Sep 09 12:45:00.187876 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.187849    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"]
Sep 09 12:45:00.201108 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.201083    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"
Sep 09 12:45:00.453980 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.453948    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"]
Sep 09 12:45:00.456671 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:45:00.456641    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b609a08_4a03_4c8f_9848_0f0068347176.slice/crio-4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf WatchSource:0}: Error finding container 4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf: Status 404 returned error can't find the container with id 4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf
Sep 09 12:45:00.487664 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.487506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"]
Sep 09 12:45:00.489986 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:45:00.489963    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77562dfb_8fd1_4714_8586_bc0eeeea0e1c.slice/crio-1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de WatchSource:0}: Error finding container 1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de: Status 404 returned error can't find the container with id 1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de
Sep 09 12:45:00.588938 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.588908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq" event={"ID":"77562dfb-8fd1-4714-8586-bc0eeeea0e1c","Type":"ContainerStarted","Data":"1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de"}
Sep 09 12:45:00.590266 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:00.590217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw" event={"ID":"6b609a08-4a03-4c8f-9848-0f0068347176","Type":"ContainerStarted","Data":"4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf"}
Sep 09 12:45:01.602984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:01.602602    2556 generic.go:356] "Generic (PLEG): container finished" podID="77562dfb-8fd1-4714-8586-bc0eeeea0e1c" containerID="a38da9ec7a0a24151cec87f50863f8757ef271f6bb2cc43ff0588b6ec945e1d5" exitCode=0
Sep 09 12:45:01.602984 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:01.602734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq" event={"ID":"77562dfb-8fd1-4714-8586-bc0eeeea0e1c","Type":"ContainerDied","Data":"a38da9ec7a0a24151cec87f50863f8757ef271f6bb2cc43ff0588b6ec945e1d5"}
Sep 09 12:45:01.605252 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:01.605184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw" event={"ID":"6b609a08-4a03-4c8f-9848-0f0068347176","Type":"ContainerStarted","Data":"92e10e568beef156377acda2a74f8a03b8c2a735c5d2a3bf3a85484f30a3dcc4"}
Sep 09 12:45:01.636856 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:01.636792    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw" podStartSLOduration=1.6367760850000002 podStartE2EDuration="1.636776085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:45:01.635565944 +0000 UTC m=+2024.266153722" watchObservedRunningTime="2026-09-09 12:45:01.636776085 +0000 UTC m=+2024.267363860"
Sep 09 12:45:02.611742 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:02.611706    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b609a08-4a03-4c8f-9848-0f0068347176" containerID="92e10e568beef156377acda2a74f8a03b8c2a735c5d2a3bf3a85484f30a3dcc4" exitCode=0
Sep 09 12:45:02.612149 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:02.611794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw" event={"ID":"6b609a08-4a03-4c8f-9848-0f0068347176","Type":"ContainerDied","Data":"92e10e568beef156377acda2a74f8a03b8c2a735c5d2a3bf3a85484f30a3dcc4"}
Sep 09 12:45:02.748041 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:02.748017    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"
Sep 09 12:45:03.617579 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:03.617550    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"
Sep 09 12:45:03.617983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:03.617592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq" event={"ID":"77562dfb-8fd1-4714-8586-bc0eeeea0e1c","Type":"ContainerDied","Data":"1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de"}
Sep 09 12:45:03.617983 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:03.617637    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1604d9fe0c7c3eee581f505f17f490ee5546a27bffa396d17ed269edf2e5b1de"
Sep 09 12:45:03.745551 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:03.745529    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"
Sep 09 12:45:04.623410 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:04.623380    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"
Sep 09 12:45:04.623911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:04.623382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw" event={"ID":"6b609a08-4a03-4c8f-9848-0f0068347176","Type":"ContainerDied","Data":"4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf"}
Sep 09 12:45:04.623911 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:04.623505    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4245254b63f69ac6f363ffecd8d839bcdad722bd942a4434e36935c86807abcf"
Sep 09 12:45:14.360336 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360295    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"]
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360766    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b609a08-4a03-4c8f-9848-0f0068347176" containerName="cleanup"
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360784    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b609a08-4a03-4c8f-9848-0f0068347176" containerName="cleanup"
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360819    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77562dfb-8fd1-4714-8586-bc0eeeea0e1c" containerName="cleanup"
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360828    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="77562dfb-8fd1-4714-8586-bc0eeeea0e1c" containerName="cleanup"
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360915    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="77562dfb-8fd1-4714-8586-bc0eeeea0e1c" containerName="cleanup"
Sep 09 12:45:14.360926 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.360926    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b609a08-4a03-4c8f-9848-0f0068347176" containerName="cleanup"
Sep 09 12:45:14.364186 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.364166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:14.366648 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.366626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:45:14.366769 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.366629    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:45:14.371086 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.371059    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"]
Sep 09 12:45:14.484620 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.484580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z\" (UID: \"1da091c0-eb75-428a-8251-25a14194192a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:14.585367 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.585331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z\" (UID: \"1da091c0-eb75-428a-8251-25a14194192a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:14.585940 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.585917    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z\" (UID: \"1da091c0-eb75-428a-8251-25a14194192a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:14.677426 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.677335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:14.946566 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:14.946490    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"]
Sep 09 12:45:14.949643 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:45:14.949615    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da091c0_eb75_428a_8251_25a14194192a.slice/crio-594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222 WatchSource:0}: Error finding container 594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222: Status 404 returned error can't find the container with id 594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222
Sep 09 12:45:15.689375 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:15.689147    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z" event={"ID":"1da091c0-eb75-428a-8251-25a14194192a","Type":"ContainerStarted","Data":"5ee59dc5d3d2a326afae790c49962c327655235f63568e5ef655c3a165248230"}
Sep 09 12:45:15.689791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:15.689406    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z" event={"ID":"1da091c0-eb75-428a-8251-25a14194192a","Type":"ContainerStarted","Data":"594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222"}
Sep 09 12:45:15.711870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:15.711820    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z" podStartSLOduration=1.711803693 podStartE2EDuration="1.711803693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:45:15.709704106 +0000 UTC m=+2038.340291881" watchObservedRunningTime="2026-09-09 12:45:15.711803693 +0000 UTC m=+2038.342391469"
Sep 09 12:45:16.695758 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:16.695721    2556 generic.go:356] "Generic (PLEG): container finished" podID="1da091c0-eb75-428a-8251-25a14194192a" containerID="5ee59dc5d3d2a326afae790c49962c327655235f63568e5ef655c3a165248230" exitCode=0
Sep 09 12:45:16.695758 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:16.695762    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z" event={"ID":"1da091c0-eb75-428a-8251-25a14194192a","Type":"ContainerDied","Data":"5ee59dc5d3d2a326afae790c49962c327655235f63568e5ef655c3a165248230"}
Sep 09 12:45:17.839076 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:17.839048    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:17.914666 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:17.914627    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca\") pod \"1da091c0-eb75-428a-8251-25a14194192a\" (UID: \"1da091c0-eb75-428a-8251-25a14194192a\") "
Sep 09 12:45:17.914987 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:17.914964    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1da091c0-eb75-428a-8251-25a14194192a" (UID: "1da091c0-eb75-428a-8251-25a14194192a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:18.016074 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:18.016033    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da091c0-eb75-428a-8251-25a14194192a-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:18.706678 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:18.706589    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z" event={"ID":"1da091c0-eb75-428a-8251-25a14194192a","Type":"ContainerDied","Data":"594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222"}
Sep 09 12:45:18.706678 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:18.706619    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"
Sep 09 12:45:18.706678 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:18.706651    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="594f83018bb6fa79633c7aa4081327e016843dc96322d6cd6adebef5e80a5222"
Sep 09 12:45:25.069962 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.069926    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:45:25.070542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.070173    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" podUID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" containerName="payload-processing" containerID="cri-o://61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571" gracePeriod=30
Sep 09 12:45:25.083635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.083595    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:45:25.083863 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.083831    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" podUID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" containerName="payload-pre-processing" containerID="cri-o://7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3" gracePeriod=30
Sep 09 12:45:25.271722 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.271690    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"]
Sep 09 12:45:25.276004 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.275978    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8ec58b-82t6z"]
Sep 09 12:45:25.357829 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.357807    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:45:25.360121 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.360102    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:45:25.476830 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.476798    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzwmg\" (UniqueName: \"kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg\") pod \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") "
Sep 09 12:45:25.477028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.476891    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plkh8\" (UniqueName: \"kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8\") pod \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") "
Sep 09 12:45:25.477028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.476921    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume\") pod \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\" (UID: \"67ee1aed-f0b0-4808-991f-d4fbd37749a4\") "
Sep 09 12:45:25.477028 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.476953    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume\") pod \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\" (UID: \"e1be66d4-1a63-45e2-abdb-7a18ce706ff9\") "
Sep 09 12:45:25.477326 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.477299    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67ee1aed-f0b0-4808-991f-d4fbd37749a4" (UID: "67ee1aed-f0b0-4808-991f-d4fbd37749a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:25.477420 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.477352    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1be66d4-1a63-45e2-abdb-7a18ce706ff9" (UID: "e1be66d4-1a63-45e2-abdb-7a18ce706ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:25.479210 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.479183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8" (OuterVolumeSpecName: "kube-api-access-plkh8") pod "67ee1aed-f0b0-4808-991f-d4fbd37749a4" (UID: "67ee1aed-f0b0-4808-991f-d4fbd37749a4"). InnerVolumeSpecName "kube-api-access-plkh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:25.479340 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.479226    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg" (OuterVolumeSpecName: "kube-api-access-xzwmg") pod "e1be66d4-1a63-45e2-abdb-7a18ce706ff9" (UID: "e1be66d4-1a63-45e2-abdb-7a18ce706ff9"). InnerVolumeSpecName "kube-api-access-xzwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:25.577589 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.577546    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwmg\" (UniqueName: \"kubernetes.io/projected/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-kube-api-access-xzwmg\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:25.577589 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.577579    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkh8\" (UniqueName: \"kubernetes.io/projected/67ee1aed-f0b0-4808-991f-d4fbd37749a4-kube-api-access-plkh8\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:25.577589 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.577591    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee1aed-f0b0-4808-991f-d4fbd37749a4-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:25.577822 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.577605    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1be66d4-1a63-45e2-abdb-7a18ce706ff9-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:25.742497 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.742459    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" containerID="7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3" exitCode=0
Sep 09 12:45:25.742683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.742509    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"
Sep 09 12:45:25.742683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.742543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" event={"ID":"e1be66d4-1a63-45e2-abdb-7a18ce706ff9","Type":"ContainerDied","Data":"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"}
Sep 09 12:45:25.742683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.742596    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw" event={"ID":"e1be66d4-1a63-45e2-abdb-7a18ce706ff9","Type":"ContainerDied","Data":"b61d89adaa61d7a01166cd756ae15ad4188bc67ddf89d256cc2b869b98e1cd54"}
Sep 09 12:45:25.742683 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.742620    2556 scope.go:117] "RemoveContainer" containerID="7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"
Sep 09 12:45:25.743913 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.743894    2556 generic.go:356] "Generic (PLEG): container finished" podID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" containerID="61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571" exitCode=0
Sep 09 12:45:25.743987 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.743933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" event={"ID":"67ee1aed-f0b0-4808-991f-d4fbd37749a4","Type":"ContainerDied","Data":"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"}
Sep 09 12:45:25.744032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.743991    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2" event={"ID":"67ee1aed-f0b0-4808-991f-d4fbd37749a4","Type":"ContainerDied","Data":"d521e9756242cefc79390635f01a535ca3478a5c40adb299c814c9f5f123194e"}
Sep 09 12:45:25.744032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.743946    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"
Sep 09 12:45:25.752562 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.752544    2556 scope.go:117] "RemoveContainer" containerID="7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"
Sep 09 12:45:25.752858 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:45:25.752839    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3\": container with ID starting with 7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3 not found: ID does not exist" containerID="7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"
Sep 09 12:45:25.752945 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.752869    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3"} err="failed to get container status \"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3\": rpc error: code = NotFound desc = could not find container \"7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3\": container with ID starting with 7bb5dba9793fd8f42700472fd1b57c17830652ac785415c20427ba75e5ff43a3 not found: ID does not exist"
Sep 09 12:45:25.752945 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.752886    2556 scope.go:117] "RemoveContainer" containerID="61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"
Sep 09 12:45:25.763971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.763951    2556 scope.go:117] "RemoveContainer" containerID="61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"
Sep 09 12:45:25.764342 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:45:25.764312    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571\": container with ID starting with 61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571 not found: ID does not exist" containerID="61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"
Sep 09 12:45:25.764411 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.764355    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571"} err="failed to get container status \"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571\": rpc error: code = NotFound desc = could not find container \"61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571\": container with ID starting with 61db01595c18cda5d00302b341abb9ac860f695fcf9e9e188378fa73f2d29571 not found: ID does not exist"
Sep 09 12:45:25.783624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.783591    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:45:25.809011 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.808979    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8ec58b-99488889d-qsfgw"]
Sep 09 12:45:25.827790 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.827766    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:45:25.837267 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.837233    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8ec58b-7564bc5f7c-xrvg2"]
Sep 09 12:45:25.901663 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.901633    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da091c0-eb75-428a-8251-25a14194192a" path="/var/lib/kubelet/pods/1da091c0-eb75-428a-8251-25a14194192a/volumes"
Sep 09 12:45:25.901990 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.901976    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" path="/var/lib/kubelet/pods/67ee1aed-f0b0-4808-991f-d4fbd37749a4/volumes"
Sep 09 12:45:25.902320 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:25.902307    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" path="/var/lib/kubelet/pods/e1be66d4-1a63-45e2-abdb-7a18ce706ff9/volumes"
Sep 09 12:45:32.745748 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.745717    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"]
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746029    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" containerName="payload-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746040    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" containerName="payload-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746063    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da091c0-eb75-428a-8251-25a14194192a" containerName="revoke-keys"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746069    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da091c0-eb75-428a-8251-25a14194192a" containerName="revoke-keys"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746078    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" containerName="payload-pre-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746084    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" containerName="payload-pre-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746132    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ee1aed-f0b0-4808-991f-d4fbd37749a4" containerName="payload-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746141    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1be66d4-1a63-45e2-abdb-7a18ce706ff9" containerName="payload-pre-processing"
Sep 09 12:45:32.746167 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.746147    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da091c0-eb75-428a-8251-25a14194192a" containerName="revoke-keys"
Sep 09 12:45:32.750431 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.750410    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:32.753018 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.752997    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:45:32.753138 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.752997    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:45:32.757025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.757003    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"]
Sep 09 12:45:32.843654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.843613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4\" (UID: \"6f812754-9286-4216-9dc2-d97e2f5138ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:32.944990 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.944953    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4\" (UID: \"6f812754-9286-4216-9dc2-d97e2f5138ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:32.945626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:32.945604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4\" (UID: \"6f812754-9286-4216-9dc2-d97e2f5138ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:33.062319 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:33.062217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:33.352882 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:33.348128    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"]
Sep 09 12:45:33.354599 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:45:33.354570    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f812754_9286_4216_9dc2_d97e2f5138ee.slice/crio-1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80 WatchSource:0}: Error finding container 1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80: Status 404 returned error can't find the container with id 1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80
Sep 09 12:45:33.785198 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:33.785158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4" event={"ID":"6f812754-9286-4216-9dc2-d97e2f5138ee","Type":"ContainerStarted","Data":"a8be5de98f82347ef7f3c89cb449d42feadf7b1f9a1b5a2c43649de93dad62e4"}
Sep 09 12:45:33.785599 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:33.785223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4" event={"ID":"6f812754-9286-4216-9dc2-d97e2f5138ee","Type":"ContainerStarted","Data":"1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80"}
Sep 09 12:45:33.800671 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:33.800624    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4" podStartSLOduration=1.8006084329999998 podStartE2EDuration="1.800608433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:45:33.799667455 +0000 UTC m=+2056.430255231" watchObservedRunningTime="2026-09-09 12:45:33.800608433 +0000 UTC m=+2056.431196205"
Sep 09 12:45:34.791057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:34.791020    2556 generic.go:356] "Generic (PLEG): container finished" podID="6f812754-9286-4216-9dc2-d97e2f5138ee" containerID="a8be5de98f82347ef7f3c89cb449d42feadf7b1f9a1b5a2c43649de93dad62e4" exitCode=0
Sep 09 12:45:34.791455 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:34.791108    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4" event={"ID":"6f812754-9286-4216-9dc2-d97e2f5138ee","Type":"ContainerDied","Data":"a8be5de98f82347ef7f3c89cb449d42feadf7b1f9a1b5a2c43649de93dad62e4"}
Sep 09 12:45:35.926740 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:35.926710    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:36.074682 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.074581    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca\") pod \"6f812754-9286-4216-9dc2-d97e2f5138ee\" (UID: \"6f812754-9286-4216-9dc2-d97e2f5138ee\") "
Sep 09 12:45:36.074940 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.074915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f812754-9286-4216-9dc2-d97e2f5138ee" (UID: "6f812754-9286-4216-9dc2-d97e2f5138ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:36.176114 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.176073    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f812754-9286-4216-9dc2-d97e2f5138ee-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:36.802765 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.802735    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"
Sep 09 12:45:36.802944 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.802767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4" event={"ID":"6f812754-9286-4216-9dc2-d97e2f5138ee","Type":"ContainerDied","Data":"1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80"}
Sep 09 12:45:36.802944 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:36.802816    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d814986c8f8ad58fa4060451cfbfe2f0a00e3b28b84c55991665a0437e4bc80"
Sep 09 12:45:43.230477 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.230382    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"]
Sep 09 12:45:43.235341 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.235313    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-679f45-29815965-dxkjw"]
Sep 09 12:45:43.267025 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.266986    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:45:43.267339 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.267297    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" podUID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" containerName="payload-pre-processing" containerID="cri-o://aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8" gracePeriod=30
Sep 09 12:45:43.445964 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.445934    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"]
Sep 09 12:45:43.449091 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.449064    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-679f45-nqzx4"]
Sep 09 12:45:43.521621 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.521592    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:45:43.641434 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.641397    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7x2\" (UniqueName: \"kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2\") pod \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") "
Sep 09 12:45:43.641642 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.641479    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume\") pod \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\" (UID: \"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0\") "
Sep 09 12:45:43.641811 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.641786    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" (UID: "990fb1c7-d48d-4d73-a8c9-49fc5f4574f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:43.643756 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.643732    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2" (OuterVolumeSpecName: "kube-api-access-th7x2") pod "990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" (UID: "990fb1c7-d48d-4d73-a8c9-49fc5f4574f0"). InnerVolumeSpecName "kube-api-access-th7x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:45:43.742377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.742338    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7x2\" (UniqueName: \"kubernetes.io/projected/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-kube-api-access-th7x2\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:43.742377 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.742368    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:43.836032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.835948    2556 generic.go:356] "Generic (PLEG): container finished" podID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" containerID="aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8" exitCode=0
Sep 09 12:45:43.836032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.835994    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"
Sep 09 12:45:43.836304 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.836038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" event={"ID":"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0","Type":"ContainerDied","Data":"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"}
Sep 09 12:45:43.836304 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.836094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p" event={"ID":"990fb1c7-d48d-4d73-a8c9-49fc5f4574f0","Type":"ContainerDied","Data":"05cdccfed528861a6386499226209a3d594ccd39d0c7bbcd57553692916c5bc6"}
Sep 09 12:45:43.836304 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.836123    2556 scope.go:117] "RemoveContainer" containerID="aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"
Sep 09 12:45:43.845069 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.845054    2556 scope.go:117] "RemoveContainer" containerID="aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"
Sep 09 12:45:43.845377 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:45:43.845358    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8\": container with ID starting with aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8 not found: ID does not exist" containerID="aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"
Sep 09 12:45:43.845428 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.845385    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8"} err="failed to get container status \"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8\": rpc error: code = NotFound desc = could not find container \"aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8\": container with ID starting with aa03cabf197a9e5183165fcbb9253b764e845e0734d41b139ac8fd8dd66ee8f8 not found: ID does not exist"
Sep 09 12:45:43.857741 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.857721    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:45:43.864388 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.864367    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-679f45-567b778474-x7n4p"]
Sep 09 12:45:43.901432 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.901404    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b609a08-4a03-4c8f-9848-0f0068347176" path="/var/lib/kubelet/pods/6b609a08-4a03-4c8f-9848-0f0068347176/volumes"
Sep 09 12:45:43.901720 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.901708    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f812754-9286-4216-9dc2-d97e2f5138ee" path="/var/lib/kubelet/pods/6f812754-9286-4216-9dc2-d97e2f5138ee/volumes"
Sep 09 12:45:43.904645 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:43.902763    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" path="/var/lib/kubelet/pods/990fb1c7-d48d-4d73-a8c9-49fc5f4574f0/volumes"
Sep 09 12:45:51.187076 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187032    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"]
Sep 09 12:45:51.187612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187523    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f812754-9286-4216-9dc2-d97e2f5138ee" containerName="revoke-keys"
Sep 09 12:45:51.187612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187543    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f812754-9286-4216-9dc2-d97e2f5138ee" containerName="revoke-keys"
Sep 09 12:45:51.187612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187557    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" containerName="payload-pre-processing"
Sep 09 12:45:51.187612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187566    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" containerName="payload-pre-processing"
Sep 09 12:45:51.187834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187689    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f812754-9286-4216-9dc2-d97e2f5138ee" containerName="revoke-keys"
Sep 09 12:45:51.187834 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.187704    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="990fb1c7-d48d-4d73-a8c9-49fc5f4574f0" containerName="payload-pre-processing"
Sep 09 12:45:51.190706 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.190684    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:51.193043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.193018    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:45:51.193043 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.193032    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sk2jd\""
Sep 09 12:45:51.197626 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.197599    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"]
Sep 09 12:45:51.208070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.208046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn\" (UID: \"14a4706a-5978-408e-b9c0-b66cc5898d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:51.309288 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.309221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn\" (UID: \"14a4706a-5978-408e-b9c0-b66cc5898d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:51.309904 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.309877    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn\" (UID: \"14a4706a-5978-408e-b9c0-b66cc5898d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:51.503747 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.503713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:51.775816 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.775787    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"]
Sep 09 12:45:51.778183 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:45:51.778146    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a4706a_5978_408e_b9c0_b66cc5898d91.slice/crio-741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e WatchSource:0}: Error finding container 741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e: Status 404 returned error can't find the container with id 741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e
Sep 09 12:45:51.877309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:51.877271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn" event={"ID":"14a4706a-5978-408e-b9c0-b66cc5898d91","Type":"ContainerStarted","Data":"741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e"}
Sep 09 12:45:52.883655 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:52.883548    2556 generic.go:356] "Generic (PLEG): container finished" podID="14a4706a-5978-408e-b9c0-b66cc5898d91" containerID="9d061cd04f41d02229fc7aaf8f946918d5c96661cceb043fd0735f5387503099" exitCode=0
Sep 09 12:45:52.884017 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:52.883645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn" event={"ID":"14a4706a-5978-408e-b9c0-b66cc5898d91","Type":"ContainerDied","Data":"9d061cd04f41d02229fc7aaf8f946918d5c96661cceb043fd0735f5387503099"}
Sep 09 12:45:54.022448 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.022423    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:54.033962 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.033933    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca\") pod \"14a4706a-5978-408e-b9c0-b66cc5898d91\" (UID: \"14a4706a-5978-408e-b9c0-b66cc5898d91\") "
Sep 09 12:45:54.034331 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.034308    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14a4706a-5978-408e-b9c0-b66cc5898d91" (UID: "14a4706a-5978-408e-b9c0-b66cc5898d91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:45:54.134573 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.134522    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4706a-5978-408e-b9c0-b66cc5898d91-maas-api-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:45:54.895529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.895502    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"
Sep 09 12:45:54.895529 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.895520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn" event={"ID":"14a4706a-5978-408e-b9c0-b66cc5898d91","Type":"ContainerDied","Data":"741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e"}
Sep 09 12:45:54.895825 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:45:54.895562    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="741df6e510084f00ec57be085b5e42f6b433e819f690aba64c84c2aa66784a3e"
Sep 09 12:46:01.698426 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.698391    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"]
Sep 09 12:46:01.701553 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.701526    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cfcdfd-29815965-76zjq"]
Sep 09 12:46:01.742383 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.742353    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:46:01.742606 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.742563    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" podUID="0856ffa0-8108-4e71-be97-630e2b5ec701" containerName="payload-processing" containerID="cri-o://6c5fe01d664c56c5f640b16ebce830de3c2c15c009b4ee9ec26ee0e5de94df33" gracePeriod=30
Sep 09 12:46:01.754633 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.754595    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:46:01.754831 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.754806    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" podUID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" containerName="payload-pre-processing" containerID="cri-o://95309e9a1d5dc8f65870596958e4ff7e93ecfff72f04c20e4ede0a73b9c037a3" gracePeriod=30
Sep 09 12:46:01.902064 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.902028    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77562dfb-8fd1-4714-8586-bc0eeeea0e1c" path="/var/lib/kubelet/pods/77562dfb-8fd1-4714-8586-bc0eeeea0e1c/volumes"
Sep 09 12:46:01.923276 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.923198    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"]
Sep 09 12:46:01.925825 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.925800    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cfcdfd-2qdgn"]
Sep 09 12:46:01.932430 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.932399    2556 generic.go:356] "Generic (PLEG): container finished" podID="0856ffa0-8108-4e71-be97-630e2b5ec701" containerID="6c5fe01d664c56c5f640b16ebce830de3c2c15c009b4ee9ec26ee0e5de94df33" exitCode=0
Sep 09 12:46:01.932571 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.932498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" event={"ID":"0856ffa0-8108-4e71-be97-630e2b5ec701","Type":"ContainerDied","Data":"6c5fe01d664c56c5f640b16ebce830de3c2c15c009b4ee9ec26ee0e5de94df33"}
Sep 09 12:46:01.934266 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.934222    2556 generic.go:356] "Generic (PLEG): container finished" podID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" containerID="95309e9a1d5dc8f65870596958e4ff7e93ecfff72f04c20e4ede0a73b9c037a3" exitCode=0
Sep 09 12:46:01.934384 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:01.934267    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" event={"ID":"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa","Type":"ContainerDied","Data":"95309e9a1d5dc8f65870596958e4ff7e93ecfff72f04c20e4ede0a73b9c037a3"}
Sep 09 12:46:02.027355 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.027329    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:46:02.029927 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.029905    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:46:02.100791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.100759    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume\") pod \"0856ffa0-8108-4e71-be97-630e2b5ec701\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") "
Sep 09 12:46:02.100999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.100810    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh9b6\" (UniqueName: \"kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6\") pod \"0856ffa0-8108-4e71-be97-630e2b5ec701\" (UID: \"0856ffa0-8108-4e71-be97-630e2b5ec701\") "
Sep 09 12:46:02.100999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.100858    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbvnj\" (UniqueName: \"kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj\") pod \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") "
Sep 09 12:46:02.100999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.100903    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume\") pod \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\" (UID: \"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa\") "
Sep 09 12:46:02.101184 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.101149    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0856ffa0-8108-4e71-be97-630e2b5ec701" (UID: "0856ffa0-8108-4e71-be97-630e2b5ec701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:46:02.101312 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.101294    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0856ffa0-8108-4e71-be97-630e2b5ec701-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:46:02.101367 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.101294    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" (UID: "1b3852d1-dcc4-4df3-8c9e-2944f477a0fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:46:02.103152 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.103130    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6" (OuterVolumeSpecName: "kube-api-access-zh9b6") pod "0856ffa0-8108-4e71-be97-630e2b5ec701" (UID: "0856ffa0-8108-4e71-be97-630e2b5ec701"). InnerVolumeSpecName "kube-api-access-zh9b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:46:02.103203 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.103157    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj" (OuterVolumeSpecName: "kube-api-access-sbvnj") pod "1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" (UID: "1b3852d1-dcc4-4df3-8c9e-2944f477a0fa"). InnerVolumeSpecName "kube-api-access-sbvnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:46:02.201942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.201886    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh9b6\" (UniqueName: \"kubernetes.io/projected/0856ffa0-8108-4e71-be97-630e2b5ec701-kube-api-access-zh9b6\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:46:02.201942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.201934    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvnj\" (UniqueName: \"kubernetes.io/projected/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-kube-api-access-sbvnj\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:46:02.201942 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.201943    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa-plugins-config-volume\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:46:02.940225 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.940198    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"
Sep 09 12:46:02.940654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.940190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx" event={"ID":"0856ffa0-8108-4e71-be97-630e2b5ec701","Type":"ContainerDied","Data":"5c3f50c748be2590214687e2931c99301f381804bd2b5b7cb3e4e82980f89f61"}
Sep 09 12:46:02.940654 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.940307    2556 scope.go:117] "RemoveContainer" containerID="6c5fe01d664c56c5f640b16ebce830de3c2c15c009b4ee9ec26ee0e5de94df33"
Sep 09 12:46:02.941753 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.941728    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx" event={"ID":"1b3852d1-dcc4-4df3-8c9e-2944f477a0fa","Type":"ContainerDied","Data":"71d79b80e2208b6b64c8033063d2699254d6cc6b758ad62b32190b6190ec82ad"}
Sep 09 12:46:02.941870 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.941783    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"
Sep 09 12:46:02.949791 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.949774    2556 scope.go:117] "RemoveContainer" containerID="95309e9a1d5dc8f65870596958e4ff7e93ecfff72f04c20e4ede0a73b9c037a3"
Sep 09 12:46:02.968491 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.968467    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:46:02.976612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.976590    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cfcdfd-5445b654d5-984kx"]
Sep 09 12:46:02.998059 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:02.998036    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:46:03.009124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:03.009056    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cfcdfd-896fb84d6-v7psx"]
Sep 09 12:46:03.901111 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:03.901073    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0856ffa0-8108-4e71-be97-630e2b5ec701" path="/var/lib/kubelet/pods/0856ffa0-8108-4e71-be97-630e2b5ec701/volumes"
Sep 09 12:46:03.901437 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:03.901423    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a4706a-5978-408e-b9c0-b66cc5898d91" path="/var/lib/kubelet/pods/14a4706a-5978-408e-b9c0-b66cc5898d91/volumes"
Sep 09 12:46:03.901737 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:03.901726    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" path="/var/lib/kubelet/pods/1b3852d1-dcc4-4df3-8c9e-2944f477a0fa/volumes"
Sep 09 12:46:17.943130 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:17.943103    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:46:17.944802 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:17.944776    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:46:17.947446 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:17.947427    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:46:17.949071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:46:17.949050    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:47:18.399111 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:47:18.398988    2556 scope.go:117] "RemoveContainer" containerID="0ab8bf76c9d2b7cd881864298c923f5ee8d38cc2ba9e24e0065a639bd11b929f"
Sep 09 12:50:18.428850 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:18.428763    2556 scope.go:117] "RemoveContainer" containerID="7e1d07fbd0e02f975227828475180021465d2bc251dc563e0370f3eef49ed1d6"
Sep 09 12:50:37.808792 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.808757    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809072    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a4706a-5978-408e-b9c0-b66cc5898d91" containerName="revoke-keys"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809086    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a4706a-5978-408e-b9c0-b66cc5898d91" containerName="revoke-keys"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809098    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" containerName="payload-pre-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809103    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" containerName="payload-pre-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809122    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0856ffa0-8108-4e71-be97-630e2b5ec701" containerName="payload-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809128    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0856ffa0-8108-4e71-be97-630e2b5ec701" containerName="payload-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809197    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0856ffa0-8108-4e71-be97-630e2b5ec701" containerName="payload-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809207    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3852d1-dcc4-4df3-8c9e-2944f477a0fa" containerName="payload-pre-processing"
Sep 09 12:50:37.809284 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.809215    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a4706a-5978-408e-b9c0-b66cc5898d91" containerName="revoke-keys"
Sep 09 12:50:37.811999 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.811981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.814307 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.814284    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a470f520-kserve-self-signed-certs\""
Sep 09 12:50:37.822062 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.822037    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:50:37.872306 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.872501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.872501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.872501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872483    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.872501 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872500    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95wl\" (UniqueName: \"kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.872674 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.872524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.973815 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974031 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974031 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974031 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974031 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973921    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95wl\" (UniqueName: \"kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974031 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.973939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974356 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.974325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974418 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.974368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.974418 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.974400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.976270 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.976226    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.976510 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.976492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:37.982150 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:37.982127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95wl\" (UniqueName: \"kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl\") pod \"e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") " pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:38.122095 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:38.121999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:38.393338 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:38.393315    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:50:38.395566 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:50:38.395536    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9dc3a0_53a5_4b71_9523_a80792ff69f8.slice/crio-ef2765f63d4ab02961619b2d6f08526aa3cf1104424abfeae197ec87aedf3789 WatchSource:0}: Error finding container ef2765f63d4ab02961619b2d6f08526aa3cf1104424abfeae197ec87aedf3789: Status 404 returned error can't find the container with id ef2765f63d4ab02961619b2d6f08526aa3cf1104424abfeae197ec87aedf3789
Sep 09 12:50:39.193449 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:39.193416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" event={"ID":"0d9dc3a0-53a5-4b71-9523-a80792ff69f8","Type":"ContainerStarted","Data":"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"}
Sep 09 12:50:39.193832 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:39.193458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" event={"ID":"0d9dc3a0-53a5-4b71-9523-a80792ff69f8","Type":"ContainerStarted","Data":"ef2765f63d4ab02961619b2d6f08526aa3cf1104424abfeae197ec87aedf3789"}
Sep 09 12:50:39.211108 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:39.211051    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" podStartSLOduration=2.211033914 podStartE2EDuration="2.211033914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:50:39.208850053 +0000 UTC m=+2361.839437827" watchObservedRunningTime="2026-09-09 12:50:39.211033914 +0000 UTC m=+2361.841621689"
Sep 09 12:50:48.122352 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:48.122307    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:48.122871 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:48.122381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:48.135341 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:48.135315    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:50:48.244536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:50:48.244503    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:51:17.970972 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:17.970943    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:51:17.973139 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:17.973114    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:51:17.975604 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:17.975583    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:51:17.977552 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:17.977529    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snxqt_dd313698-c628-4132-8a0e-bbcc55063c11/ovn-acl-logging/0.log"
Sep 09 12:51:18.448078 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:18.447993    2556 scope.go:117] "RemoveContainer" containerID="5ee59dc5d3d2a326afae790c49962c327655235f63568e5ef655c3a165248230"
Sep 09 12:51:18.457451 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:18.457430    2556 scope.go:117] "RemoveContainer" containerID="92e10e568beef156377acda2a74f8a03b8c2a735c5d2a3bf3a85484f30a3dcc4"
Sep 09 12:51:18.465200 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:18.465172    2556 scope.go:117] "RemoveContainer" containerID="a38da9ec7a0a24151cec87f50863f8757ef271f6bb2cc43ff0588b6ec945e1d5"
Sep 09 12:51:27.850176 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:27.850139    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:51:42.871526 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:42.871416    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" podUID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" containerName="main" containerID="cri-o://fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a" gracePeriod=45
Sep 09 12:51:43.128102 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.128026    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:51:43.268382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268382 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268392    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268647 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268416    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268647 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268441    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268647 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z95wl\" (UniqueName: \"kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268647 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268496    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs\") pod \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\" (UID: \"0d9dc3a0-53a5-4b71-9523-a80792ff69f8\") "
Sep 09 12:51:43.268838 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268715    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home" (OuterVolumeSpecName: "home") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:43.268838 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268725    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:43.268838 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.268746    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:43.270917 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.270874    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl" (OuterVolumeSpecName: "kube-api-access-z95wl") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "kube-api-access-z95wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:51:43.270917 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.270883    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:51:43.271118 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.270933    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm" (OuterVolumeSpecName: "dshm") pod "0d9dc3a0-53a5-4b71-9523-a80792ff69f8" (UID: "0d9dc3a0-53a5-4b71-9523-a80792ff69f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 12:51:43.370060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370017    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95wl\" (UniqueName: \"kubernetes.io/projected/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-kube-api-access-z95wl\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.370060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370052    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tls-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.370060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370062    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-dshm\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.370060 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370071    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-home\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.370388 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370078    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-model-cache\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.370388 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.370086    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9dc3a0-53a5-4b71-9523-a80792ff69f8-tmp-dir\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:51:43.485447 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.485414    2556 generic.go:356] "Generic (PLEG): container finished" podID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" containerID="fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a" exitCode=0
Sep 09 12:51:43.485605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.485468    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"
Sep 09 12:51:43.485605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.485504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" event={"ID":"0d9dc3a0-53a5-4b71-9523-a80792ff69f8","Type":"ContainerDied","Data":"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"}
Sep 09 12:51:43.485605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.485555    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw" event={"ID":"0d9dc3a0-53a5-4b71-9523-a80792ff69f8","Type":"ContainerDied","Data":"ef2765f63d4ab02961619b2d6f08526aa3cf1104424abfeae197ec87aedf3789"}
Sep 09 12:51:43.485605 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.485576    2556 scope.go:117] "RemoveContainer" containerID="fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"
Sep 09 12:51:43.495090 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.495071    2556 scope.go:117] "RemoveContainer" containerID="fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"
Sep 09 12:51:43.495453 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:51:43.495432    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a\": container with ID starting with fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a not found: ID does not exist" containerID="fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"
Sep 09 12:51:43.495542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.495470    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a"} err="failed to get container status \"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a\": rpc error: code = NotFound desc = could not find container \"fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a\": container with ID starting with fd5deefeeb294b069b2debe2bb9d8867446c07423dc6b3622e7cc96363d7035a not found: ID does not exist"
Sep 09 12:51:43.511903 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.511863    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:51:43.521329 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.521293    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a470f520-kserve-79557958c9-vz8xw"]
Sep 09 12:51:43.902231 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:43.902136    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" path="/var/lib/kubelet/pods/0d9dc3a0-53a5-4b71-9523-a80792ff69f8/volumes"
Sep 09 12:51:53.188097 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.188052    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:51:53.188612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.188396    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" containerName="main"
Sep 09 12:51:53.188612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.188407    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" containerName="main"
Sep 09 12:51:53.188612 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.188481    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9dc3a0-53a5-4b71-9523-a80792ff69f8" containerName="main"
Sep 09 12:51:53.191186 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.191167    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.193298 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.193277    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 12:51:53.194175 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.194148    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 12:51:53.194330 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.194312    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 12:51:53.194330 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.194297    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qz8xf\""
Sep 09 12:51:53.194460 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.194332    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 12:51:53.203635 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.203595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:51:53.360271 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.360206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.360483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.360291    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.360483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.360348    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.360483 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.360370    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59ml\" (UniqueName: \"kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.461361 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.461325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.461542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.461376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.461542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.461415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.461542 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.461431    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59ml\" (UniqueName: \"kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.462087 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.462062    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.464111 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.464088    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.464197 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.464110    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.473920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.473890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59ml\" (UniqueName: \"kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml\") pod \"maas-api-6c5d9d8b4c-dcdv5\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") " pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.502032 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.501991    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:53.775301 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.775272    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:51:53.777019 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:51:53.776986    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff937028_0837_419a_bec1_b59388dfff2e.slice/crio-a93f9f7d60fe82b1165f3e5c327e2feb0934b1d7acfdafc5b8853f186d4ba6bc WatchSource:0}: Error finding container a93f9f7d60fe82b1165f3e5c327e2feb0934b1d7acfdafc5b8853f186d4ba6bc: Status 404 returned error can't find the container with id a93f9f7d60fe82b1165f3e5c327e2feb0934b1d7acfdafc5b8853f186d4ba6bc
Sep 09 12:51:53.778358 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:53.778339    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 12:51:54.537614 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:54.537570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" event={"ID":"ff937028-0837-419a-bec1-b59388dfff2e","Type":"ContainerStarted","Data":"a93f9f7d60fe82b1165f3e5c327e2feb0934b1d7acfdafc5b8853f186d4ba6bc"}
Sep 09 12:51:55.544309 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:55.544276    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" event={"ID":"ff937028-0837-419a-bec1-b59388dfff2e","Type":"ContainerStarted","Data":"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"}
Sep 09 12:51:55.544709 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:55.544418    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:51:55.563082 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:51:55.563024    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" podStartSLOduration=0.998605625 podStartE2EDuration="2.56300741s" totalImagesPullingTime="1.564401785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:51:55.560685741 +0000 UTC m=+2438.191273544" watchObservedRunningTime="2026-09-09 12:51:55.56300741 +0000 UTC m=+2438.193595250"
Sep 09 12:52:00.534931 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.534891    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:52:00.535457 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.535127    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" podUID="ff937028-0837-419a-bec1-b59388dfff2e" containerName="maas-api" containerID="cri-o://38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f" gracePeriod=30
Sep 09 12:52:00.540071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.540046    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:52:00.776838 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.776813    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:52:00.821774 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.821690    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t59ml\" (UniqueName: \"kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml\") pod \"ff937028-0837-419a-bec1-b59388dfff2e\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") "
Sep 09 12:52:00.821774 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.821733    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs\") pod \"ff937028-0837-419a-bec1-b59388dfff2e\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") "
Sep 09 12:52:00.821774 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.821754    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca\") pod \"ff937028-0837-419a-bec1-b59388dfff2e\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") "
Sep 09 12:52:00.822126 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.821781    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls\") pod \"ff937028-0837-419a-bec1-b59388dfff2e\" (UID: \"ff937028-0837-419a-bec1-b59388dfff2e\") "
Sep 09 12:52:00.822190 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.822116    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff937028-0837-419a-bec1-b59388dfff2e" (UID: "ff937028-0837-419a-bec1-b59388dfff2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 12:52:00.824124 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.824101    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff937028-0837-419a-bec1-b59388dfff2e" (UID: "ff937028-0837-419a-bec1-b59388dfff2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:52:00.824214 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.824152    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff937028-0837-419a-bec1-b59388dfff2e" (UID: "ff937028-0837-419a-bec1-b59388dfff2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 12:52:00.824270 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.824233    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml" (OuterVolumeSpecName: "kube-api-access-t59ml") pod "ff937028-0837-419a-bec1-b59388dfff2e" (UID: "ff937028-0837-419a-bec1-b59388dfff2e"). InnerVolumeSpecName "kube-api-access-t59ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 12:52:00.922985 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.922940    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59ml\" (UniqueName: \"kubernetes.io/projected/ff937028-0837-419a-bec1-b59388dfff2e-kube-api-access-t59ml\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:52:00.922985 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.922981    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-metrics-certs\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:52:00.922985 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.922991    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff937028-0837-419a-bec1-b59388dfff2e-openshift-service-ca\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:52:00.923227 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:00.923002    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff937028-0837-419a-bec1-b59388dfff2e-maas-api-tls\") on node \"ip-10-0-128-117.ec2.internal\" DevicePath \"\""
Sep 09 12:52:01.578830 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.578794    2556 generic.go:356] "Generic (PLEG): container finished" podID="ff937028-0837-419a-bec1-b59388dfff2e" containerID="38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f" exitCode=0
Sep 09 12:52:01.578830 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.578835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" event={"ID":"ff937028-0837-419a-bec1-b59388dfff2e","Type":"ContainerDied","Data":"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"}
Sep 09 12:52:01.579390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.578845    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"
Sep 09 12:52:01.579390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.578909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5" event={"ID":"ff937028-0837-419a-bec1-b59388dfff2e","Type":"ContainerDied","Data":"a93f9f7d60fe82b1165f3e5c327e2feb0934b1d7acfdafc5b8853f186d4ba6bc"}
Sep 09 12:52:01.579390 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.578931    2556 scope.go:117] "RemoveContainer" containerID="38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"
Sep 09 12:52:01.588071 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.588049    2556 scope.go:117] "RemoveContainer" containerID="38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"
Sep 09 12:52:01.588410 ip-10-0-128-117 kubenswrapper[2556]: E0909 12:52:01.588389    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f\": container with ID starting with 38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f not found: ID does not exist" containerID="38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"
Sep 09 12:52:01.588460 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.588427    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f"} err="failed to get container status \"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f\": rpc error: code = NotFound desc = could not find container \"38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f\": container with ID starting with 38e0786c75489ee58c5cff53a198c5c83d8a59a121819efd36d6212dea61b78f not found: ID does not exist"
Sep 09 12:52:01.601140 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.601108    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:52:01.605568 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.605544    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c5d9d8b4c-dcdv5"]
Sep 09 12:52:01.902000 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:01.901914    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff937028-0837-419a-bec1-b59388dfff2e" path="/var/lib/kubelet/pods/ff937028-0837-419a-bec1-b59388dfff2e/volumes"
Sep 09 12:52:18.488702 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:18.488670    2556 scope.go:117] "RemoveContainer" containerID="a8be5de98f82347ef7f3c89cb449d42feadf7b1f9a1b5a2c43649de93dad62e4"
Sep 09 12:52:18.496676 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:18.496651    2556 scope.go:117] "RemoveContainer" containerID="9d061cd04f41d02229fc7aaf8f946918d5c96661cceb043fd0735f5387503099"
Sep 09 12:52:24.441767 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:24.441732    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f95fd98c-5zm89_975d5213-a56b-4b59-8d4b-8a629a1fcdc1/authorino/0.log"
Sep 09 12:52:30.880535 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:30.880505    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj_dc097444-537d-4563-923d-34b4bfb29a60/copy-manifests/0.log"
Sep 09 12:52:30.887652 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:30.887620    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj_dc097444-537d-4563-923d-34b4bfb29a60/manager/0.log"
Sep 09 12:52:32.504738 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:32.504708    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f95fd98c-5zm89_975d5213-a56b-4b59-8d4b-8a629a1fcdc1/authorino/0.log"
Sep 09 12:52:32.859895 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:32.859817    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rg5k6_5d231dc9-5ee7-4417-a9fe-0fd147ab0246/kuadrant-console-plugin/0.log"
Sep 09 12:52:33.963907 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:33.963849    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b9df8cfb-ncf29_a6f971be-c0d0-4649-91ec-e87a40aea4f5/kube-auth-proxy/0.log"
Sep 09 12:52:34.204036 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:34.204003    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xj6c6_19cafb7c-bac5-4114-b0f4-d709037af1bc/payload-pre-processing/0.log"
Sep 09 12:52:34.445874 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:34.445797    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-867b7bd764-5qsj7_63cc97ea-6218-4078-b500-bec5db94a85f/router/0.log"
Sep 09 12:52:35.049624 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:35.049589    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-4775t_ccb5f1fc-0747-4b10-953e-5a90445533bf/main/0.log"
Sep 09 12:52:35.295685 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:35.295657    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd94qr_4f3657b2-a1c6-4a7e-830e-081774bf803a/main/0.log"
Sep 09 12:52:35.531708 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:35.531676    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jfgs2_b569646c-fd4e-4000-8336-ebdd37054075/main/0.log"
Sep 09 12:52:40.857103 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.857063    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ffhw/must-gather-hnwmj"]
Sep 09 12:52:40.857522 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.857405    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff937028-0837-419a-bec1-b59388dfff2e" containerName="maas-api"
Sep 09 12:52:40.857522 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.857416    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff937028-0837-419a-bec1-b59388dfff2e" containerName="maas-api"
Sep 09 12:52:40.857522 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.857467    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff937028-0837-419a-bec1-b59388dfff2e" containerName="maas-api"
Sep 09 12:52:40.859681 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.859665    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:40.862070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.862038    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5ffhw\"/\"default-dockercfg-xxfxj\""
Sep 09 12:52:40.862070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.862054    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ffhw\"/\"kube-root-ca.crt\""
Sep 09 12:52:40.862872 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.862855    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ffhw\"/\"openshift-service-ca.crt\""
Sep 09 12:52:40.868499 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.868477    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ffhw/must-gather-hnwmj"]
Sep 09 12:52:40.977070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.977035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7g8\" (UniqueName: \"kubernetes.io/projected/fb236e26-c0d8-43eb-b105-a5db13cde741-kube-api-access-cz7g8\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:40.977285 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:40.977084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb236e26-c0d8-43eb-b105-a5db13cde741-must-gather-output\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.077759 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.077725    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7g8\" (UniqueName: \"kubernetes.io/projected/fb236e26-c0d8-43eb-b105-a5db13cde741-kube-api-access-cz7g8\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.077759 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.077775    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb236e26-c0d8-43eb-b105-a5db13cde741-must-gather-output\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.078070 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.078054    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb236e26-c0d8-43eb-b105-a5db13cde741-must-gather-output\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.087965 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.087931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7g8\" (UniqueName: \"kubernetes.io/projected/fb236e26-c0d8-43eb-b105-a5db13cde741-kube-api-access-cz7g8\") pod \"must-gather-hnwmj\" (UID: \"fb236e26-c0d8-43eb-b105-a5db13cde741\") " pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.169227 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.169134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ffhw/must-gather-hnwmj"
Sep 09 12:52:41.440033 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.440002    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ffhw/must-gather-hnwmj"]
Sep 09 12:52:41.442576 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:52:41.442547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb236e26_c0d8_43eb_b105_a5db13cde741.slice/crio-30e06fd18a744deb19b9074188eea3168c4d4264e638925eeae811bd6c0533b1 WatchSource:0}: Error finding container 30e06fd18a744deb19b9074188eea3168c4d4264e638925eeae811bd6c0533b1: Status 404 returned error can't find the container with id 30e06fd18a744deb19b9074188eea3168c4d4264e638925eeae811bd6c0533b1
Sep 09 12:52:41.758311 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:41.758269    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ffhw/must-gather-hnwmj" event={"ID":"fb236e26-c0d8-43eb-b105-a5db13cde741","Type":"ContainerStarted","Data":"30e06fd18a744deb19b9074188eea3168c4d4264e638925eeae811bd6c0533b1"}
Sep 09 12:52:42.766037 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:42.766000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ffhw/must-gather-hnwmj" event={"ID":"fb236e26-c0d8-43eb-b105-a5db13cde741","Type":"ContainerStarted","Data":"e7f44709f4c109d90dfe777c70a5c75f1265da155f8a355ef17ab867cd9e12f9"}
Sep 09 12:52:42.766490 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:42.766047    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ffhw/must-gather-hnwmj" event={"ID":"fb236e26-c0d8-43eb-b105-a5db13cde741","Type":"ContainerStarted","Data":"3480808ef447c398fd92581380fc187d8e2b9d31654141e4bebeb8713857bed2"}
Sep 09 12:52:42.781819 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:42.781745    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5ffhw/must-gather-hnwmj" podStartSLOduration=1.9788656690000002 podStartE2EDuration="2.78172522s" totalImagesPullingTime="802.859551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:52:42.780514791 +0000 UTC m=+2485.411102567" watchObservedRunningTime="2026-09-09 12:52:42.78172522 +0000 UTC m=+2485.412312997"
Sep 09 12:52:43.989873 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:43.989840    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l7hxr_7c195e29-9ded-4d92-bebe-429d6354bd70/global-pull-secret-syncer/0.log"
Sep 09 12:52:44.130057 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:44.130022    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qs7x4_fd1f2f26-9ca4-4350-b0c1-e1cbcd1d972c/konnectivity-agent/0.log"
Sep 09 12:52:44.187517 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:44.187475    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-117.ec2.internal_707ed0fa4514c61dc617937a9e19c4ec/haproxy/0.log"
Sep 09 12:52:48.006448 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:48.006396    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f95fd98c-5zm89_975d5213-a56b-4b59-8d4b-8a629a1fcdc1/authorino/0.log"
Sep 09 12:52:48.170185 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:48.170157    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rg5k6_5d231dc9-5ee7-4417-a9fe-0fd147ab0246/kuadrant-console-plugin/0.log"
Sep 09 12:52:50.254920 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:50.254889    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-hcl8g_eb1847d7-9261-487a-be2c-972566991417/cluster-monitoring-operator/0.log"
Sep 09 12:52:50.531323 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:50.531212    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-krcpp_81e77ee6-53ac-462b-a314-315459b3e9a1/node-exporter/0.log"
Sep 09 12:52:50.554094 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:50.554067    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-krcpp_81e77ee6-53ac-462b-a314-315459b3e9a1/kube-rbac-proxy/0.log"
Sep 09 12:52:50.577220 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:50.577195    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-krcpp_81e77ee6-53ac-462b-a314-315459b3e9a1/init-textfile/0.log"
Sep 09 12:52:53.691584 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:53.691552    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/1.log"
Sep 09 12:52:53.697750 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:53.697708    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-247l8_3228e9fa-08dc-4baa-be44-fd7b9be2b7f6/console-operator/2.log"
Sep 09 12:52:54.286903 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:54.286880    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55c5f847b8-pfdfg_134d2744-9502-4faa-8c43-db5565cda735/console/0.log"
Sep 09 12:52:55.040306 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.040265    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"]
Sep 09 12:52:55.046669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.046638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.053908 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.053852    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"]
Sep 09 12:52:55.118793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.118757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-podres\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.118793 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.118795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-lib-modules\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.119026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.118819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2qj\" (UniqueName: \"kubernetes.io/projected/c6d525ee-04ed-498e-94b6-afa4a57c7940-kube-api-access-db2qj\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.119026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.118902    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-proc\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.119026 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.118938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-sys\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.220177 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-podres\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.220618 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-lib-modules\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.220785 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2qj\" (UniqueName: \"kubernetes.io/projected/c6d525ee-04ed-498e-94b6-afa4a57c7940-kube-api-access-db2qj\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.220945 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-proc\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.221166 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.221153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-sys\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.221321 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.221296    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-sys\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.221394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-lib-modules\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.221394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.220526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-podres\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.221394 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.221081    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d525ee-04ed-498e-94b6-afa4a57c7940-proc\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.230323 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.230291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2qj\" (UniqueName: \"kubernetes.io/projected/c6d525ee-04ed-498e-94b6-afa4a57c7940-kube-api-access-db2qj\") pod \"perf-node-gather-daemonset-ppqfl\" (UID: \"c6d525ee-04ed-498e-94b6-afa4a57c7940\") " pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.361854 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.361758    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.661020 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.660988    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"]
Sep 09 12:52:55.663578 ip-10-0-128-117 kubenswrapper[2556]: W0909 12:52:55.663543    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6d525ee_04ed_498e_94b6_afa4a57c7940.slice/crio-b877c885e65bfdd9dbb7472f52a2bc87170b23bf2281a167639910ed5adc4ab6 WatchSource:0}: Error finding container b877c885e65bfdd9dbb7472f52a2bc87170b23bf2281a167639910ed5adc4ab6: Status 404 returned error can't find the container with id b877c885e65bfdd9dbb7472f52a2bc87170b23bf2281a167639910ed5adc4ab6
Sep 09 12:52:55.851527 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.851482    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl" event={"ID":"c6d525ee-04ed-498e-94b6-afa4a57c7940","Type":"ContainerStarted","Data":"86abf065627495a357603aa8e10749775351e16a6b0c871409bd871f9bd06dcf"}
Sep 09 12:52:55.851730 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.851638    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:52:55.851730 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.851690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl" event={"ID":"c6d525ee-04ed-498e-94b6-afa4a57c7940","Type":"ContainerStarted","Data":"b877c885e65bfdd9dbb7472f52a2bc87170b23bf2281a167639910ed5adc4ab6"}
Sep 09 12:52:55.869839 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:55.869776    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl" podStartSLOduration=0.869753745 podStartE2EDuration="869.753745ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:52:55.866753607 +0000 UTC m=+2498.497341400" watchObservedRunningTime="2026-09-09 12:52:55.869753745 +0000 UTC m=+2498.500341523"
Sep 09 12:52:56.344735 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:56.344707    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5j95_0878ddec-9390-4909-b2ea-4cb57aea3698/dns/0.log"
Sep 09 12:52:56.368227 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:56.368196    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5j95_0878ddec-9390-4909-b2ea-4cb57aea3698/kube-rbac-proxy/0.log"
Sep 09 12:52:56.610011 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:56.609930    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gqw7g_4dc63ba9-7b3b-4eaf-9900-4be507b168b7/dns-node-resolver/0.log"
Sep 09 12:52:57.191969 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:57.191939    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bm5b2_5b1b1875-e67e-49de-9073-0333de341328/node-ca/0.log"
Sep 09 12:52:58.797059 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:58.797031    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b9df8cfb-ncf29_a6f971be-c0d0-4649-91ec-e87a40aea4f5/kube-auth-proxy/0.log"
Sep 09 12:52:58.867656 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:58.867621    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xj6c6_19cafb7c-bac5-4114-b0f4-d709037af1bc/payload-pre-processing/0.log"
Sep 09 12:52:58.926014 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:58.925977    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-867b7bd764-5qsj7_63cc97ea-6218-4078-b500-bec5db94a85f/router/0.log"
Sep 09 12:52:59.943081 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:52:59.943036    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqnpx_d79601f5-c28a-4c83-8c21-57e7e1d31668/serve-healthcheck-canary/0.log"
Sep 09 12:53:00.423536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:00.423498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-75fzc_640e5fd6-5c1f-4b28-a14d-4cb59e947583/insights-operator/1.log"
Sep 09 12:53:00.423898 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:00.423882    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-75fzc_640e5fd6-5c1f-4b28-a14d-4cb59e947583/insights-operator/0.log"
Sep 09 12:53:00.512670 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:00.512646    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mdwth_012a3b9a-7354-42df-b198-2011231deceb/kube-rbac-proxy/0.log"
Sep 09 12:53:00.534783 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:00.534733    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mdwth_012a3b9a-7354-42df-b198-2011231deceb/exporter/0.log"
Sep 09 12:53:00.557492 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:00.557459    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mdwth_012a3b9a-7354-42df-b198-2011231deceb/extractor/0.log"
Sep 09 12:53:01.870971 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:01.870938    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5ffhw/perf-node-gather-daemonset-ppqfl"
Sep 09 12:53:03.733188 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:03.733122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj_dc097444-537d-4563-923d-34b4bfb29a60/manager/0.log"
Sep 09 12:53:03.762669 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:03.762640    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9bf5b7c6-gbwvj_dc097444-537d-4563-923d-34b4bfb29a60/copy-manifests/0.log"
Sep 09 12:53:04.904936 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:04.904909    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b9df8cfb-ncf29_a6f971be-c0d0-4649-91ec-e87a40aea4f5/kube-auth-proxy/0.log"
Sep 09 12:53:04.960608 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:04.960501    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xj6c6_19cafb7c-bac5-4114-b0f4-d709037af1bc/payload-pre-processing/0.log"
Sep 09 12:53:05.006718 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:05.006643    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-867b7bd764-5qsj7_63cc97ea-6218-4078-b500-bec5db94a85f/router/0.log"
Sep 09 12:53:05.293078 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:05.292999    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-df65498f9-v22bh_c01f7f7f-9746-4f81-9437-db2ed9639f21/manager/0.log"
Sep 09 12:53:07.138416 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:07.138340    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b9df8cfb-ncf29_a6f971be-c0d0-4649-91ec-e87a40aea4f5/kube-auth-proxy/0.log"
Sep 09 12:53:07.197536 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:07.197505    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xj6c6_19cafb7c-bac5-4114-b0f4-d709037af1bc/payload-pre-processing/0.log"
Sep 09 12:53:07.237690 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:07.237665    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-867b7bd764-5qsj7_63cc97ea-6218-4078-b500-bec5db94a85f/router/0.log"
Sep 09 12:53:10.882516 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:10.882423    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xwht7_0ef14022-b373-4895-8eff-b16e98fa7029/kube-storage-version-migrator-operator/1.log"
Sep 09 12:53:10.883153 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:10.883135    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xwht7_0ef14022-b373-4895-8eff-b16e98fa7029/kube-storage-version-migrator-operator/0.log"
Sep 09 12:53:11.927608 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:11.927577    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6fjrr_13e8d794-ed88-440f-b2fe-e5f36f718e5b/kube-multus/0.log"
Sep 09 12:53:12.121017 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.120935    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/kube-multus-additional-cni-plugins/0.log"
Sep 09 12:53:12.143668 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.143637    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/egress-router-binary-copy/0.log"
Sep 09 12:53:12.165644 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.165611    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/cni-plugins/0.log"
Sep 09 12:53:12.189560 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.189527    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/bond-cni-plugin/0.log"
Sep 09 12:53:12.211548 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.211508    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/routeoverride-cni/0.log"
Sep 09 12:53:12.234344 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.234317    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/whereabouts-cni-bincopy/0.log"
Sep 09 12:53:12.257089 ip-10-0-128-117 kubenswrapper[2556]: I0909 12:53:12.257062    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6xx4_956af29a-af32-4d5a-8eaa-ce00d3b27996/whereabouts-cni/0.log"